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Preface 


This  RAND  Corporation  monograph  is  designed  to  assist  the  Office 
of  the  Under  Secretary  of  Defense  for  Policy  (OUSD/P)  in  its  efforts 
to  develop  an  approach  for  assessing  the  effectiveness  and  efficiency  of 
Department  of  Defense  (DoD)  security  cooperation  programs  around 
the  world.  Specifically,  the  monograph  outlines  an  assessment  frame¬ 
work  that  can  enhance  OUSD/P’s  ability  to  determine  which  partner¬ 
ship  capacity  programs  and  activities  are  achieving  the  desired  effects 
as  defined  in  the  guidance  documents,  and  how.  This  assessment 
framework  is  built  around  five  themes  that  emerged  at  an  assessment 
workshop  hosted  by  RAND  in  May  2008:  setting  direction,  design¬ 
ing  assessments,  preparing  for  assessment,  conducting  assessments,  and 
explaining  assessments  to  others. 

The  monograph  will  assist  OUSD/P  in  refining  its  security  coop¬ 
eration  program  assessment  guidance  to  the  combatant  commands 
(COCOMs),  defense  agencies,  and  services.  It  is  meant  to  provide  an 
assessment  structure  for  OUSD/P,  one  specifically  useful  for  the  Build¬ 
ing  Partner  Capacity  (BPC)  program,  in  addition  to  the  broader  secu¬ 
rity  cooperation  programs  that  are  managed  directly  by  OUSD/P. 

This  research  was  sponsored  by  the  Office  of  the  Under  Secre¬ 
tary  of  Defense  for  Policy  and  was  conducted  within  the  International 
Security  and  Defense  Policy  Center  of  the  RAND  National  Defense 
Research  Institute,  a  federally  funded  research  and  development  center 
sponsored  by  the  Office  of  the  Secretary  of  Defense,  the  Joint  Staff, 
the  Unified  Combatant  Commands,  the  Department  of  the  Navy,  the 
U.S.  Marine  Corps,  the  defense  agencies,  and  the  defense  Intelligence 
Community. 


iv  A  Framework  to  Assess  Programs  for  Building  Partnerships 


For  more  information  on  RAND’s  International  Security  and 
Defense  Policy  Center,  contact  the  Director,  James  Dobbins.  He  can  be 
reached  by  email  at  James_Dobbins@rand.org;  by  phone  at  703-413- 
1100,  extension  5134;  or  by  mail  at  RAND,  1200  South  Hayes  Street, 
Arlington,  Virginia  22202-5050.  More  information  about  RAND  is 
available  at  www.rand.org. 
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Summary 


Security  cooperation  activities  conducted  by  DoD  entities  with 
other  nations’  defense  organizations  range  from  the  very  visible — 
training,  equipping,  and  exercising  together — to  those  that  are  less 
obvious,  such  as  holding  bilateral  talks,  workshops,  and  conferences 
and  providing  education.  Yet,  it  is  often  challenging  to  determine  if 
these  activities  have  contributed  to  U.S.  objectives — whether  at  the 
level  of  national  security,  department,  COCOM,  or  service — and  if 
so,  by  how  much  or  in  what  ways.  Because  security  cooperation  is  both 
dispersed  and  long-term,  it  is  difficult  to  measure.  At  present,  program 
assessments  are  inconsistent  and  of  varying  quality.  They  may  lack 
objectivity  since  they  are  frequently  conducted  by  the  organizations 
that  implement  the  programs.  A  comprehensive  framework  for  assess¬ 
ing  is  lacking,  although  efforts  are  in  place  to  remedy  that  deficiency. 
This  project  was  devoted  to  that  end. 


Key  Assessment  Themes 

RAND’s  proposed  program  assessment  framework  is  built  around 
five  themes  that  emerged  at  a  May  2008  assessment  workshop  that 
included  DoD  security  cooperation  assessment  experts,  planners,  and 
program  managers. 

1.  Setting  Direction.  An  overall  goal  or  set  of  objectives  should 
be  identified  for  security  cooperation  activities.  The  challenge 
is  that  most  programs  support  multiple  end-states  and  multiple 
stakeholders. 
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2.  Designing  Assessments.  Here  the  issues  include  assessing  both 
the  efficiency  of  the  programs  and  their  ultimate  effectiveness  as 
measured  against  specific  security  goals. 

•  Building  measurable  objectives  into  plans.  Now  that  the 
Office  of  the  Secretary  of  Defense’s  (OSD’s)  Guidance  for 
Employment  of  the  Force  (GEF)1  calls  for  program  assess¬ 
ment  against  theater  campaign  plans,  it  is  essential  to  design 
a  similar  assessment  process  across  COCOMs. 

•  Balancing  objective  and  subjective  assessments.  Outcomes- 
based  assessments  need  to  include  both  types. 

•  Measuring  the  sustainability  of  partner  capabilities  over  time. 
Too  often,  assessments  stop  with  immediate  outcomes. 

•  Fixing  inconsistent  language  and  terminology.  These  differ¬ 
ences  continue  to  hinder  development  of  standard  assessment 
frameworks  across  the  security  cooperation  community. 

3.  Preparing  for  Assessment.  The  lack  of  personnel  with  appropri¬ 
ate  assessment  backgrounds  is  a  concern,  leading  to  the  sugges¬ 
tion  of  training  some  experienced  assessment  professionals  who 
could  help  programs  and  COCOMs  alike. 

4.  Conducting  Assessments.  Workshop  participants  noted  the 
lack  of  standardized  data  across  COCOMs  and  programs,  the 
need  to  reach  out  for  data  beyond  program  managers  and  U.S. 
personnel,  and  the  complications  of  classified  information.  They 
suggested  establishing  either  special  assessment  teams  that  can 
be  sent  into  the  field  or  a  new,  dedicated  assessment  office. 

5.  Explaining  Assessments.  This  requires  both  an  understanding 
of  how  security  cooperation  programs  complement  (or  impede) 
other  efforts  of  the  military  and  other  government  agencies  and 
a  better  explanation  of  DoD  assessment  processes  to  key  audi¬ 
ences  and  stakeholders. 


1  The  GEF  replaces  the  OSD  Security  Cooperation  Guidance  and  was  signed  by  the  Secre- 
tary  of  Defense  in  May  2008. 
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Key  Elements  of  the  Assessment  Program  Framework 

At  the  highest  levels,  the  National  Security  Strategy,  the  National 
Defense  Strategy,  the  National  Military  Strategy,  the  GEF,  and  the 
COCOM  theater  campaign  plans  form  the  basis  for  strategy-driven 
security  cooperation  and  are  key  documents  in  selecting  priority  part¬ 
ners  of  interest  to  the  United  States.  The  military  departments  imple¬ 
ment  OSD  guidance  and  support  the  COCOM  theater  campaign  plans 
by  developing  their  own  campaign  support  plans.  In  thinking  about 
assessment,  stakeholders  are  critical.  For  the  purposes  of  this  analysis, 
we  defined  principal  stakeholders  as  those  that  are  involved  with  over¬ 
seeing,  planning,  or  implementing/executing  security  cooperation  pro¬ 
grams.  In  terms  of  authorities ,  Title  10  of  the  U.S.  Code  serves  as  the 
primary  authority  for  many  DoD  security  cooperation  programs.  Title 
22  provides  the  basic  authority  for  the  State  Department  to  oversee 
and  resource  security  assistance  programs,  even  though  they  are  mostly 
administered  by  DoD. 

In  making  assessments,  three  types  of  indicators  are  critical: 

•  Inputs  are  the  resources — both  money  and  manpower — that  are 
required  to  execute  an  event. 

•  Outputs  are  the  direct  products  of  an  event,  activity,  or  program. 

•  Outcomes  are  the  effect  of  outputs  on  the  target  audience,  or 
changes  in  program  participants’  behavior,  knowledge,  skills, 
status  and/or  level  of  functioning. 

Metrics  are  observable,  measurable  evidence  of  outputs  and  outcomes. 

These  elements  can  be  combined  in  a  “hierarchy  of  evaluation” 
that  contains  five  levels  of  assessment  (see  Figure  S.l).1 2 

Fevel  1  is  the  foundation.  Here,  evaluation  focuses  on  the  prob¬ 
lem  to  be  solved  or  goal  to  be  met,  the  population  to  be  served,  and  the 
kinds  of  services  that  might  contribute  to  a  solution.  Fevel  2  addresses 
the  design  of  a  policy  or  program  and  seeks  to  confirm  that  what  was 
planned  is  adequate  to  achieve  the  desired  objectives.  Fevel  3  asks 


1  The  term  hierarchy  of  evaluation  comes  from  Richard  A.  Berk  and  Peter  H.  Rossi,  Think - 

ing  About  Program  Evaluation ,  Newbury  Park,  Calif.:  Sage  Publications,  1990. 
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Figure  S.1 

A  Hierarchy  of  Evaluation 


Level  5 

Level  4 

Level  3 

Level  2 

Level  1 


Assessment  of  cost-effectiveness 
I  Assessment  of  outputs  and  outcomes 
Assessment  of  process  and  implementation 
Assessment  of  design  and  theory 
Assessment  of  need  for  program 
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whether  execution  met  the  design  at  Level  2.  At  Level  4,  outputs  are 
the  products  of  program  activities,  and  outcomes  are  the  changes  that 
result.  This  is  the  first  level  of  assessment  at  which  solutions  to  the 
problem  that  originally  motivated  the  program  can  be  seen.  Finally,  at 
Level  5,  the  assessment  looks  across  programs  for  cost-effectiveness,  or 
“bang  for  the  buck.” 


Assessment  Functions 

At  each  level,  four  functional  assessment  roles  must  be  performed 
(although  the  same  organization  may  perform  multiple  roles):  data  col¬ 
lector,  assessor,  reviewer,  and  integrator.  The  data  collector  is  responsible 
for  gathering  all  relevant  information.  The  assessor  sets  data  collection 
standards  and  evaluates  the  program.  The  reviewer  develops  methods 
and  conducts  periodic  inspections  or  audits.  Finally,  the  integrator  is 
responsible  for  organizing  and  synthesizing  programmatic  assessments 
to  meet  DoD  requirements. 


Implementing  the  Assessment  Framework 

Given  multiple  stakeholders,  these  levels  of  assessment  help  to  determine 
assessment  roles  in  the  same  way  as  stakeholders’  authority  influences 
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their  responsibilities,  which  in  turn  shape  the  sets  of  decisions  that  fall 
within  their  respective  purviews.  In  general,  the  highest  levels  of  lead¬ 
ership  within  DoD,  typically  based  on  recommendations  by  OUSD/P, 
will  be  responsible  for  the  critical  decisions  about  needs  for  programs 
and  their  design,  as  well  as  for  assessing  cost-effectiveness  across  pro¬ 
grams.  The  study  team  recommends  that  service  and  COCOM  stake¬ 
holders  should  primarily  concentrate  on  matters  of  process  and  imple¬ 
mentation,  as  well  as  on  outcomes  and  impacts. 

Many  DoD  organizations  might  serve  as  data  collectors,  asses¬ 
sors,  reviewers,  and  integrators  for  OUSD/P-managed  security  cooper¬ 
ation  programs.  Table  S.l.  represents  RAND’s  notional  and  proposed 
organizational  assignments  as  a  first  step  in  establishing  an  integrated 
structure  of  assessment  roles  and  responsibilities  for  DoD.  A  key  goal  is 
to  inject  a  higher  level  of  objectivity  into  the  assessment  process  and  to 
move  away  from  the  current  self-assessment  approach. 


Table  S.l 

Assessment  Roles  for  OUSD/P-Managed  Programs 


Assessment 

Decision 

Data 

Collector 

Assessor 

Reviewer 

Integrator 

Level  1: 

Need  for 
program 

OUSD/P 
Partnership 
Strategy  (PS)/ 
program 
manager 

OUSD/P  PS 

OSD  Program 
Analysis  and 
Evaluation 
(PA&E) 

OUSD/P  PS/DSCA 
via  BPC  portfolio 
manager  (BP 
CPM) 

Level  2: 
Design/theory 

Program 

manager 

Program 

manager 

New  OSD  SC 
auditor? 

OUSD/P  PS/DSCA 
(via  BP  CPM) 

Level  3: 

Process/ 

implementation 

Program 

manager 

Program 

manager 

New  OSD  SC 
auditor? 

OUSD/P  PS/DSCA 
(via  BP  CPM) 

Level  4: 

Outputs/ 

outcomes 

Program 

manager 

OSD  PS 

New  OSD  SC 
auditor? 

OUSD/P  PS/DSCA 
(via  BP  CPM) 

Level  5: 

Cost- 

effectiveness 

Program 

manager 

OSD  PS 

OSD  PA&E 

OUSD/P  PS/DSCA 
(via  BP  CPM) 

NOTES:  SC  =  security  cooperation;  DSCA  =  Defense  Security  Cooperation  Agency. 
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A  Proposed  OUSD/P  Assessment  Approach 

RAND’s  proposed  approach  seeks  to  deal  with  the  many  obstacles  to 
measuring  the  effectiveness  of  BPC  and  broader  security  cooperation 
programs,  with  the  aim  of  producing  assessments  that  are  consistent 
with  the  spirit  and  intent  of  the  GEF.  It  is  meant  to  provide  a  useful 
assessment  structure  for  OUSD/P — specifically  for  the  BPC  and  other 
security  cooperation  programs  that  are  managed  directly  by  OUSD/P. 
It  is  also  designed  to  encourage  objective  analysis. 

We  propose  that  OUSD/P  consider  taking  four  basic  steps: 

1.  Coordinate  with  the  other  major  security  cooperation  players 
to  reach  a  consensus  regarding  the  definitions  of,  and  linkages 
among,  the  key  assessment  framework  elements. 

2.  Clarify  the  assessment  roles  and  responsibilities  of  each  program 
stakeholder  by  level  of  assessment. 

3.  Approve  appropriate  assessment  questions  for  each  level  of 
assessment. 

4.  Implement  a  comprehensive  security  cooperation  assessment 
framework  for  the  programs  that  OUSD/P  directly  manages. 


Recommendations 

To  help  OUSD/P  and  DSCA  adopt  a  BPC  and,  more  broadly,  a  secu¬ 
rity  cooperation  assessment  framework  focused  on  programs,  we  lay 
out  the  following  specific  recommendations  for  implementing  the 
assessment  framework  in  relation  to  the  five  themes  articulated  in  this 
monograph. 

Setting  Direction 

•  Work  with  key  DoD  stakeholders  to  clarify  program  assessment 
responsibilities  in  the  GEF. 

•  Build  enduring  security  cooperation  goals  and  measurable  pro¬ 
gram  objectives. 

•  Clarify  and  standardize  assessment  terminology. 


Summary  xix 


•  Ensure  that  OUSD/Policy  and  DSCA  are  the  assessment  integra¬ 
tors  for  programs  they  manage. 

Designing  Assessment  Processes 

•  Leverage  assessment  processes  that  already  exist. 

•  Within  programs,  determine  stakeholder  authorities  and  roles 
among  the  different  levels  of  assessment. 

•  Consider  an  assessment  function  within  OUSD/Policy  to  help 
implement  the  assessment  framework  for  OUSD/P-managed 
programs. 

Preparing  Stakeholders  for  Assessment 

•  OUSD/Policy  and  OUSD/Personnel  and  Readiness  could  work 
with  the  Defense  Institute  for  Security  Assistance  Management 
and  DSCA  to  develop  a  professional  curriculum  for  security 
cooperation  assessments. 

Conducting  Assessments 

•  Specify  stakeholder  data  collection  roles  and  responsibilities. 

•  Explore  external  indicators. 

•  Standardize  assessment  questions  within  and  across  programs. 

•  Develop  a  feedback  loop  for  “setting  direction.” 

•  Consider  a  pilot  program  to  test  the  assessment  framework. 

Explaining  Assessment  Results 

•  Develop  clearer  linkages  between  assessment  and  planning. 

•  Use  results  to  inform  decisions  about  programs. 

•  Identify  examples  where  multiple,  coordinated  programs  have 
achieved  desired  effects. 

•  Consider  using  the  Global  Theater  Security  Cooperation  Man¬ 
agement  Information  System  to  store  program  assessments. 
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Introduction 


The  Department  of  Defense  (DoD)  long  has  worked  with  allies  and 
partners  in  a  security  cooperation  context  to  build  strong  and  endur¬ 
ing  relationships,  reinforce  others’  capacity  both  to  defend  themselves 
and  to  work  in  coalitions,  and  ensure  U.S.  access  to  foreign  territo¬ 
ries  for  operational  purposes.  The  activities  conducted  by  DoD  entities 
range  from  the  very  visible — training,  equipping,  and  exercising  with 
others — to  those  that  are  less  obvious,  such  as  holding  bilateral  talks, 
workshops,  and  conferences  and  providing  education.  Yet,  it  is  often 
challenging  to  specify  just  how  these  activities  have  contributed  to  U.S. 
objectives — whether  at  the  level  of  national  security,  department,  com¬ 
batant  command  (COCOM),  or  service — and  how  much  or  in  what 
ways.  Because  security  cooperation  is  both  dispersed  and  long-term, 
a  comprehensive  framework  for  assessing  it  is  still  lacking,  although 
efforts  are  in  place  to  remedy  that  deficiency. 

Assessments  are  important  at  all  levels.  At  the  highest  level,  the 
Office  of  the  Undersecretary  of  Defense  for  Policy  (OUSD/P)  is  work¬ 
ing  to  develop  an  assessment  framework  for  the  key  security  coopera¬ 
tion  programs  it  manages,  a  framework  that  might  then  serve  as  an 
example  for  the  rest  of  DoD.  The  project  on  which  this  monograph 
reports  is  part  of  that  effort.  For  OUSD/P,  assessments  support  deci¬ 
sions  about  the  programs  it  manages,  provide  a  basis  for  comparing 
their  effectiveness  and  efficiency,  offer  insights  into  improving  them  if 
issues  are  discovered,  and  suggest  new  programs  for  DoD  when  gaps 
are  identified. 
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Those  who  plan  and  execute  security  cooperation  intuitively  know 
whether  they  have  gained  ground  with  the  respective  partner  nations  as 
a  result  of  individual  programs  and  activities.  That  fact  came  through 
loud  and  clear  in  many  focused  discussions  conducted  by  the  RAND 
project  team.  It  also  was  a  consistent  theme  during  the  workshop  held 
at  RAND  in  May  2008,  which  sought  to  bring  together  program  man¬ 
agers  and  program  assessment  officials  from  the  combatant  commands, 
services,  and  other  key  agencies.  At  the  most  basic  level,  officials  often 
assert  that  the  relationship  with  the  partner  country  is  simply  “better” 
than  it  was  prior  to  the  execution  of  the  activity. 

Although  the  assertions  appear  to  ring  true,  it  is  more  difficult  to 
validate  this  general  sense  of  accomplishment  empirically,  especially  to 
audiences  in  higher  headquarters  and  in  Congress.  At  present,  assess¬ 
ments  are  largely  conducted  by  the  executing  organization.  They  are 
thus  subject,  no  matter  how  carefully  done,  to  suspicions  about  the  self- 
interests  of  the  assessors.  Moreover,  self-assessment  is  even  less  convinc¬ 
ing  when  it  is  done  by  program  managers,  planners,  and  executers  who 
often  rotate  in  and  out  of  positions  rapidly — in  as  little  as  a  year  for 
military  officers.  They  are  not  likely  to  be  regarded  as  having  the  long¬ 
term  experience  necessary  to  understand  and  evaluate  the  effectiveness 
of  a  program  in  a  given  country  over  time. 

This  monograph  will  assist  the  Office  of  the  Secretary  of  Defense 
in  its  efforts  to  develop  an  approach  for  assessing  the  effectiveness  and 
efficiency  of  DoD  security  cooperation  programs  around  the  world. 
Specifically,  the  monograph  outlines  an  assessment  framework  that  can 
enhance  OUSD/P’s  ability  to  determine  which  programs  and  activities 
are  achieving  the  desired  effects  as  defined  in  the  guidance  documents, 
and  why.  The  monograph  will  assist  OUSD/P  in  refining  its  program 
assessment  guidance  to  the  COCOMs,  defense  agencies,  and  services  at 
the  program  level.  It  is  meant  to  provide  a  structure  for  assessments — 
in  particular,  for  Building  Partner  Capacity  (BPC)  and  broader  secu¬ 
rity  cooperation  programs  that  OUSD/P  manages  directly.  Finally,  it 
is  aimed  at  injecting  greater  objectivity  into  the  current,  largely  self- 
assessment,  approach. 

While  the  focus  of  this  monograph  is  on  the  program  level  of 
security  cooperation,  the  authors  acknowledge  the  need  for  a  national- 
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level  assessment  that  synthesizes  the  evaluations  of  all  security  coop¬ 
eration  programs.  The  United  States  seeks  to  conduct  security  coop¬ 
eration  in  a  deliberate  and  carefully  orchestrated  manner  that  takes 
into  account  relatively  enduring  U.S.  national  security  interests  arrayed 
against  an  ever-changing  international  security  environment.  Thus, 
security  cooperation  priorities  change  over  time,  and  a  national-level 
assessment  mechanism  is  necessary  to  underpin  decisions  concerning 
resource  and  program  increases,  decreases,  and  reallocations.  We  rec¬ 
ommend  an  initial  step  in  that  direction  by  suggesting  an  integrating 
role  for  the  OUSD/P.  However,  that  function  falls  somewhat  short  of 
the  comprehensive  national-level  integrated  security  cooperation  assess¬ 
ment  that  ultimately  will  be  required.  Of  course,  such  an  assessment 
should  comport  with  the  security  cooperation  priorities  delineated  in 
the  Guidance  for  Employment  of  the  Force  (GEF). 


Defining  Key  Terminology  for  Security  Cooperation 

Key  terms  that  are  used  throughout  this  monograph  require  an  expla¬ 
nation.  Security  cooperation  and  its  subset,  security  assistance ,  have  a 
long  history.  According  to  the  Defense  Security  Cooperation  Agency 
(DSCA)  Web  site,  security  cooperation  includes  “those  activities  con¬ 
ducted  with  allies  and  friendly  nations  to:  build  relationships  that  pro¬ 
mote  specified  U.S.  interests,  build  allied  and  friendly  nation  capabili¬ 
ties  for  self-defense  and  coalition  operations,  [and]  provide  U.S.  forces 
with  peacetime  and  contingency  access.”1  Examples  include  training 
and  combined  exercises,  operational  meetings,  contacts  and  exchanges, 
security  assistance,  medical  and  engineering  team  engagements,  coop- 


1  See  DSCA,  Frequently  Asked  Questions  (FAQs),  2009.  For  a  complete  discussion  of  Title 
10  and  Title  22  authorities,  as  well  as  a  general  description  of  security  cooperation  and  secu¬ 
rity  assistance,  see  Defense  Institute  of  Security  Assistance  Management,  DISAM’s  Online 
Green  Book,  2008. 
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erative  development,  acquisition  and  technical  interchanges,  and  scien¬ 
tific  and  technology  collaboration.2 

Security  assistance  is  a  subset  of  security  cooperation  and  con¬ 
sists  of  “a  group  of  programs,  authorized  by  law  that  allows  the  trans¬ 
fer  of  military  articles  and  services  to  friendly  foreign  governments.”3 
Examples  of  these  programs  include  Foreign  Military  Sales  (FMS), 
Foreign  Military  Financing  (FMF),  International  Military  Education 
and  Training  (IMET),  and  Direct  Commercial  Sales. 

Building  Partnership  Capacity  is  another  key  term  used  through¬ 
out  this  monograph.  The  2006  Quadrennial  Defense  Review  (QDR) 
and  the  2006  BPC  Execution  Roadmap  emphasize  the  importance  of 
building  the  security  and  defense  capabilities  of  partner  countries  that 
will  enable  them  to  make  valuable  contributions  to  coalition  opera¬ 
tions  and  improve  their  own  indigenous  capabilities.4  Building  partner¬ 
ship  capacity  is  a  term  of  art  employed  to  describe  “targeted  efforts  to 
improve  the  collective  capabilities  and  performance  of  the  Department 
of  Defense  and  its  partners.”5  BPC  can  be  thought  of  as  an  umbrella 
objective  that  draws  on  the  elements  of  security  cooperation.  The  pri¬ 
mary  goal  of  BPC  is  to  implement  a  multiagency  approach  to  meeting 
U.S.  strategic  objectives,  one  that  includes  not  only  U.S.  government 
entities  but  also  key  partners  and  allies  abroad.  At  its  best,  BPC  tends 


2  Department  of  the  Air  Force,  Air  Force  Security  Cooperation  Strategy:  Building  Capacity, 
Integrating  Capabilities,  unclassified  excerpt  from  classified  document,  Washington,  D.C.: 
September  2006,  p.  3. 

3  Defense  Security  Cooperation  Agency  (DSCA),  “Security  Assistance  Management 
Manual  (SAMM),”  Department  of  Defense,  DoD  5105. 38-M,  2003.  A  full  list  of  security 
assistance  programs  may  be  found  on  p.  35. 

4  The  QDR  BPC  Execution  Roadmap  ( Building  Partnership  Capacity:  QDR  Execution 
Roadmap,  Washington,  D.C.,  May  2006),  published  by  The  Office  of  the  Secretary  of 
Defense  (OSD)  and  the  Joint  Staff  J-5,  is  an  evolving  concept.  It  not  only  includes  guidance 
on  how  DoD  should  train  and  equip  foreign  military  forces  but  also  discusses  the  need  to 
improve  the  capacity  of  other  security  services  (stability  police,  border  guards,  customs,  etc.) 
within  our  partner  countries.  Moreover,  the  concept  also  refers  to  the  need  to  improve  DoD’s 
ability  to  work  with  nonmilitary  forces  (e.g.,  U.S.  interagency,  nongovernmental  organiza¬ 
tions,  coalition  partners,  and  the  private  sector)  in  an  operational  context  for  integrated 
operations. 

5  Department  of  Defense,  May  2006,  p.  4. 
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to  emphasize  the  “fit”  between  U.S.  regional  objectives  and  the  capac¬ 
ity  being  built  or  expanded.  Programs  described  as  BPC  ideally  seek  to 
embrace  the  partner’s  ability  to  contribute  to  U.S.  strategic  goals.  While 
this  monograph  focuses  on  BPC-related  programs  with  a  regional  ally 
or  partner,  we  believe  that  the  assessment  approach  promulgated  in 
the  chapters  that  follow  is  also  applicable  to  the  broader  set  of  security 
cooperation  programs  that  are  also  focused  on  building  bilateral  or 
multilateral  defense  relationships. 

Other  key  terms  used  throughout  the  monograph  include  fund¬ 
ing  source,  initiative,  program,  activity,  and  event.  Funding  sources  are 
large  umbrella  resource  streams  that  fund  initiatives  or  programs.  The 
Freedom  Support  Act,  which  authorizes  resources  for  many  initia¬ 
tives  and  programs  in  Eurasia,  is  an  example  of  a  funding  source.  For 
example,  the  Freedom  Support  Act  authorizes  funding  for  the  State 
Department’s  Export  Control  and  Related  Border  Security  program. 
Initiatives  are  funding  sources  for  a  collection  of  programs  that  pursue 
a  particular  set  of  goals.  An  example  of  an  initiative  is  the  Warsaw 
Initiative  Fund  (WIF),  which  funds  programs  in  central  and  southern 
Europe  as  well  as  Eurasia.  It  is  important  to  note,  however,  that  for 
administrative  purposes,  OSD/P  treats  WIF  as  a  program. 

Program,  the  focus  of  this  monograph,  can  be  thought  of  as  a  set 
of  activities  or  events  coordinated  to  achieve  a  certain  set  of  objectives. 
At  a  minimum,  a  program  has  the  following  defining  characteristics: 

•  a  mission  and  set  of  specific  objectives 

•  activities  or  events 

•  managers)  for  policy  and/or  resource  oversight 

•  reporting  requirements  to  an  oversight  agency  or  office. 

Some  programs  have  their  own  line  items  in  the  DoD  budget 
and  therefore  do  not  have  to  solicit  funds  from  other  sources  to  exe¬ 
cute  activities.  Examples  include  the  Cooperative  Threat  Reduction 
Program,  overseen  by  OUSD/P  and  executed  mainly  by  the  Defense 
Threat  Reduction  Agency  (DTRA).  In  contrast,  other  programs  rely 
on  initiatives  or  multiple  funding  sources.  Examples  include  most  pro¬ 
grams  executed  by  the  COCOMs  and  component  commands,  such 
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as  Chairman  of  the  Joint  Chiefs  of  Staff  exercises,  funded  by  the  Joint 
Staff,  and  the  National  Guard  State  Partnership  Program  (SPP),  which 
is  executed  by  the  National  Guard  but  often  funded  out  of  traditional 
COCOM  activity  resources. 

To  complicate  matters,  different  offices  or  even  agencies  may  be 
responsible  for  the  various  functions — policy  and  planning,  resource 
management,  and  program  execution.  Examples  include  FMF,  FMS, 
and  the  IMET  programs,  all  of  which  are  executed  by  DoD  but  funded 
and  overseen  by  the  State  Department.  The  key  point  from  an  assess¬ 
ment  perspective  is  that  virtually  all  security  cooperation  programs 
have  multiple  stakeholders. 

Activities  and  events  are  actions  directed,  funded,  and/or  super¬ 
vised  by  program  managers.  Activities  are  particular  kinds  of  interac¬ 
tions  funded  by  programs,  such  as  defense  and  military  contacts  (e.g., 
army-to-army  staff  talks),  while  events  are  specific,  scheduled,  time- 
delimited  interactions  that  incorporate  U.S.  and  partner  representa¬ 
tives  (e.g.,  U.S.-United  Kingdom  army-to-army  staff  talks). 

Table  1.1  shows  the  relationship  among  funding  sources,  initia¬ 
tives,  programs,  activities,  and  events,  using  some  examples. 


Table  1.1 

Distinguishing  the  Terms 


Term 

Defining  Characteristics 

Example 

Funding  source 

Money 

Freedom  Support  Act 

Initiative 

Money  and  broad  goals 

Warsaw  Initiative  Fund 

Program 

Specific  mission/objectives,  manager, 
activities,  reporting  requirements 

OSD  Defense  and  Military 
Contacts  program 

Activity 

Specific  kinds  of  interactions  funded 
by  programs  that  include  U.S.  and 
partner  representatives;  designed 
to  address  specific  objectives 

Service-level  staff  talks 

Event 

Specific  kinds  of  activities;  may 
occur  annually  or  at  other  specified 
regular  intervals 

U.S.  army-to-army  staff 
talks  with  the  United 
Kingdom 
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Why  Assess? 

The  answer  to  this  question  is  fairly  straightforward.  Even  if  deci¬ 
sions  can  be  made  on  the  basis  of  ad  hoc  or  intuitive  judgments,  those 
decisions  are  hard  to  defend  on  those  grounds  alone.  Many  decisions 
require  assessments  based  on  more  extensive  or  rigorous  research  meth¬ 
ods:  There  are  regular  calls  for  assessments  across  most  aspects  of 
government  and  military  activity,  and  security  cooperation  is  no  excep¬ 
tion.  The  GEF,  while  elevating  the  prominence  of  security  cooperation, 
explicitly  calls  for  annual  assessments  to  be  delivered  to  OUSD/P. 

Rigorous  assessments  can  also  be  a  critical  part  of  the  dialogue 
with  Congress  in  justifying  programs.  In  addition  to  this  high-level 
call  for  security  cooperation  assessment,  security  cooperation  practi¬ 
tioners  are  well  aware  of  the  frequency  with  which  one  stakeholder  or 
another  requests  (or  imposes)  additional  assessment-related  reporting. 
Quality  assessment  of  security  cooperation  programs  will  contribute  to 
improved  decisionmaking  at  all  levels,  including  oversight,  planning, 
management,  resourcing,  and  execution. 


Who  Should  Conduct  Assessments? 

Stakeholders  who  are  involved  in  planning,  resourcing,  and  executing 
security  cooperation  programs  and  activities  all  have  some  kind  of  role 
in  the  assessment  process.  That  role  is  usually  defined  in  the  authori¬ 
ties  that  govern  the  program’s  execution.  As  laid  out  by  the  study  team, 
those  roles  include 

•  determining  the  need  for  a  program 

•  determining  objectives  (i.e.,  output  and  outcome) 

•  designing  activities 

•  controlling  resources  (e.g.,  funding,  manpower,  equipment, 
infrastructure) 

•  conducting  activities. 

A  stakeholder’s  roles  suggest  its  assessment  responsibilities.  For 
example,  if  a  stakeholder 
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•  controls  resources,  it  is  responsible  for  assessing  the  use  of  those 
resources 

•  develops  policy  outcome  goals  for  a  program,  it  assesses  outcomes 

•  develops  agency  output  goals  for  a  program,  it  assesses  outputs 

•  performs  an  administrative  or  executing  function,  it  assesses 
processes. 

The  bottom  line  is  that  understanding  what  each  organization 
is  doing  relative  to  the  programs  makes  it  possible  to  identify  what  it 
should  be  assessing.  Assessment  roles  for  stakeholders  are  discussed  in 
detail  in  Chapter  Three. 


Who  Currently  Conducts  Assessments? 

Assessments  of  DoD  BPC  and  broader  security  cooperation  activi¬ 
ties  are  conducted  presently  by  many  entities,  including  OUSD/P, 
COCOMs,  services,  defense  agencies  (such  as  DSCA),  and  defense 
support  agencies  (such  as  DTRA).  However,  as  mentioned  previously, 
these  are  self-assessments  for  the  most  part  and  are  thus  open  to  ques¬ 
tions  about  their  objectivity.  A  central  theme  of  this  analysis  is  that 
doing  better  assessments  is  both  important  and  distinctly  possible. 

For  at  least  a  decade,  country  directors  and  program  managers 
have  been  attempting  to  set  benchmarks  for  progress  that  go  beyond 
the  old  adage  of  “engaging  for  engagement’s  sake.”6  Stoplight  charts,  to 
include  red,  yellow,  and  green  indicators  and  the  input  of  knowledge¬ 
able  subject  matter  experts,  have  been  used  as  a  tool  to  help  DoD  plan¬ 
ners  assess  progress.  Most  of  these  assessments  are  quantitative  in  focus 
(e.g.,  they  count  numbers  of  events,  resources  expended),  although  there 
are  also  efforts  to  conduct  more  difficult  qualitative  assessments.  The 
emphasis  is  generally  on  the  inputs  and/or  outputs  of  specific  activities, 
rather  than  on  assessing  the  longer-term  outcomes.  The  COCOMs 
tend  to  focus  their  assessment  efforts  at  the  country  level,  while  the 


6  In  other  words,  just  because  an  activity — an  exercise,  a  training  event,  or  a  workshop,  for 
example — was  done  in  the  past  does  not  necessarily  mean  that  it  is  appropriate  to  continue 
in  the  future. 
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services  tend  to  focus  more  at  the  program  level  (although  the  services 
also  take  countries  into  consideration  through  their  country  plans). 

In  recent  years,  the  COCOMs  and  the  services  have  engaged  in 
their  own  specific  assessment  endeavors,  ranging  from  the  subjective 
stoplight  charts,  to  setting  measures  of  performance  and  effectiveness, 
to  more  qualitative  types  of  assessments.  However,  every  stakeholder 
does  assessment  differently  and  uses  different  terminology,  so  there  is 
no  common  “assessment  language”  shared  among  DoD  stakeholders. 
There  is  still  no  standard  approach  that  would  enhance  DoD’s  abil¬ 
ity  to  make  decisions  about  when  to  continue,  expand,  or  cut  existing 
resources.  Nor  is  there  a  consistent  way  to  provide  assessment  results 
to  important  external  stakeholders,  such  as  the  Office  of  Management 
and  Budget  and  Congress,  that  would  provide  the  needed  evidence  to 
show  where,  how,  and  by  how  much  DoD  security  cooperation  activi¬ 
ties  are  having  a  significant  impact. 


Why  Assess  at  the  Program  Level? 

This  analysis  is  focused  on  assessment  at  the  program  level,  for  several 
reasons.  First,  it  is  at  the  program  level  that  the  most  important  deci¬ 
sions  about  continuing,  expanding,  or  cutting  programs  and  resources 
devoted  to  security  cooperation  are  made.  Second,  assessments  at  that 
level  bring  to  bear  the  different  authorities,  roles,  and  responsibilities 
of  multiple  security  cooperation  stakeholders.  Third,  programs  provide 
insight  into  requirements  across  countries  and  regions.  For  example, 
some  DoD  stakeholders,  such  as  the  COCOMs,  tend  to  focus  on 
country-  or  regional-level  assessments.  For  them,  program  assessments 
would  provide  the  information  necessary  to  understand  the  effective¬ 
ness  of  DoD  security  cooperation  across  a  country  or  region.  In  contrast, 
OUSD/P  and  the  services  are  looking  for  insight  into  requirements  for 
security  cooperation  programs  across  COCOMs.  Finally,  a  program- 
level  assessment  can  provide  insight  into  how  well  a  given  stakeholder 
is  achieving  its  objectives  as  measured  against  strategic  goals. 
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This  rationale  was  recognized  in  the  GEF,  which  requires 
program-level  assessments  by  the  COCOMs  and  the  services.7  But 
what  is  a  program  for  assessment  purposes?  In  this  analysis,  the  RAND 
team  suggests  taking  funding  sources  for  initiatives — that  is,  collec¬ 
tions  of  programs  like  the  Warsaw  Initiative  Fund — and  separating 
them  into  their  programmatic  parts.  Thus,  OUSD/P  would  assess  the 
programs  funded  by  WIF,  such  as  the  Defense  Institution  Building,  the 
Civil  Military  Emergency  Preparedness  Program,  the  Partnership  for 
Peace  Information  Management  System,  logistics  exchanges,  and  sev¬ 
eral  other  programs.  In  security  assistance,  the  programs  to  be  assessed 
might  be  major  FMS  cases,  such  as  the  F-16  sale  to  Chile,  rather  than 
FMS  as  a  whole,  which  are  far  too  large  to  assess  effectively. 

To  be  sure,  assessment  is  important  at  other  levels  as  well,  and 
assessment  in  general  might  be  approached  as  a  set  of  building  blocks. 
For  example,  it  is  critical  for  event  planners  and  implementers  to  pro¬ 
vide  an  assessment  of  each  event  after  its  conclusion — communicated, 
ideally,  through  a  detailed  after-action  report  tied  to  specific  objectives. 
Event-level  assessment  is  critical  to  understanding  the  overall  effective¬ 
ness  of  the  program  or  programs  that  fund  those  events. 


Objectives  and  Approach 

This  monograph  develops  and  illustrates  the  key  elements  of  a 
program-focused  assessment  framework  for  the  Office  of  the  Under 
Secretary  of  Defense  for  Policy  within  the  scope  and  spirit  of  the 
Guidance  for  Employment  of  the  Force.  Given  the  basics  of  program 
assessment,  including  definitions  of  programs  and  stakeholder  roles, 
we  now  turn  to  constructing  a  program  assessment  framework  built 


7  The  RAND  team  supports  this  approach  to  security  cooperation  assessments  for  DoD 
as  a  whole,  but  we  do  not  discount  the  importance  of  other  assessments,  such  as  country- 
focused  assessments.  We  have  found  through  our  work  that  a  useful  approach  is  to  start  with 
the  program  level  and  then  consider  the  impact  of  that  program,  and  the  host  of  other  pro¬ 
grams,  in  the  respective  partner  country.  Such  an  approach  can  help  policymakers,  program 
managers,  and  country  directors  to  identify  ways  in  which  different  security  cooperation 
tools  might  be  combined  to  achieve  the  desired  effects. 
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around  the  five  themes  that  emerged  at  the  assessment  workshop  hosted 
by  RAND  in  May  2008: 

1.  Setting  direction 

•  What  kind  of  program  assessment  guidance  should  OUSD/P 
provide  to  DoD  entities? 

2.  Designing  an  assessment  framework 

•  How  should  OUSD/P  assess  its  security  cooperation  pro¬ 
grams  and  activities? 

•  What  kinds  of  assessment  questions  and  indicators  are 
appropriate? 

3.  Preparing  for  assessment 

•  How  should  OUSD/P  determine  which  security  cooperation 
stakeholders  should  conduct  the  appropriate  assessments? 

•  Specifically,  what  resources  (i.e.,  skilled  personnel)  are 
needed? 

4.  Conducting  assessments 

•  What  are  the  major  challenges  to  conducting  program 
assessments  (e.g.,  lack  of  standardized  data  across  the 
COCOMs)? 

5.  Explaining  assessments  to  others 

•  How  should  OUSD/P  report  the  results  of  assessment? 

To  answer  these  questions,  the  study  team  started  with  a  week- 
long  assessment  workshop  with  DoD  security  cooperation  assessment 
experts,  planners,  and  program  managers,  held  at  RAND  in  May 
2008.  To  develop  the  workshop  agenda  and  the  subsequent  assess¬ 
ment  framework  and  ultimately  to  help  guide  the  discussion,  the  study 
team  drew  on  its  extensive  experience  developing  security  cooperation 
assessment  frameworks  for  the  U.S.  Army,  the  U.S.  Air  Force,  and  the 
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Defense  Threat  Reduction  Agency.8  The  team  brought  to  bear  its  expe¬ 
rience  assessing  OUSD/P-overseen  security  cooperation  programs  (the 
Warsaw  Initiative  Fund,  the  Cooperative  Threat  Reduction  Program, 
and  the  National  Guard  State  Partnership  Program,  in  particular).  The 
team  also  conducted  a  literature  review  of  DoD  strategic  guidance  on 
security  cooperation  and  gained  a  thorough  understanding  of  the  six 
OUSD/P-managed  programs  for  building  partner  capacity.  OUSD/P 
selected  those  programs  as  its  initial  focus  for  assessment: 

•  Warsaw  Initiative  Fund 

•  National  Guard  State  Partnership  Program 

•  Global  Train  and  Equip  (“1206”)  Program 

•  The  Regional  Centers9 

•  Regional  Defense  Combating  Terrorism  Fellowship  Program 

•  Overseas  Humanitarian,  Disaster  and  Civic  Aid  Program. 

The  Warsaw  Initiative  Fund  supports  developing  NATO  Partner¬ 
ship  for  Peace  member  countries  in  reforming  their  defense  institu¬ 
tions,  improving  interoperability  with  the  United  States  and  NATO, 
and  integrating  further  with  NATO.  OUSD/P  Partnership  Strategy  has 


8  See  Jefferson  P.  Marquis,  Richard  E.  Darilek,  Jasen  J.  Castillo,  Cathryn  Quantic  Thur¬ 
ston,  Anny  Wong,  Cynthia  Huger,  Andrea  Mejia,  Jennifer  D.P.  Moroney,  Brian  Nichiporuk, 
and  Brett  Steele,  Assessing  the  Value  of  Army  Security  Cooperation  Activities ,  Santa  Monica, 
Calif.:  RAND  Corporation,  MG-329-A,  2006;  Jennifer  D.P.  Moroney,  Adam  Grissom, 
and  Jefferson  P.  Marquis,  A  Capabilities-Based  Strategy  for  Army  Security  Cooperation,  Santa 
Monica,  Calif.:  RAND  Corporation,  MG-563-A,  2007;  Jennifer  D.P.  Moroney,  Nancy 
E.  Blacker,  Renee  Buhr,  James  McFadden,  Cathryn  Quantic  Thurston,  and  Anny  Wong, 
Building  Partner  Capabilities  for  Coalition  Operation,  Santa  Monica,  Calif.:  RAND  Cor¬ 
poration,  MG-635-A,  2007;  Jennifer  D.P.  Moroney,  Kim  Cragin,  Eric  Gons,  Beth  Grill, 
John  E.  Peters,  and  Rachel  Swanger,  International  Cooperation  with  Partner  Air  Forces,  Santa 
Monica,  Calif:  RAND  Corporation,  MG-790-A  (forthcoming);  Jennifer  D.P.  Moroney  and 
Joe  Hogler,  with  Benjamin  Bahney,  Kim  Cragin,  David  R.  Howell,  Charlotte  Lynch,  and 
Rebecca  Zimmerman,  Building  Partner  Capacity  to  Combat  Weapons  of  Mass  Destruction, 
Santa  Monica,  Calif.:  RAND  Corporation  MG-783-DTRA  (forthcoming). 

9  The  five  DoD  Regional  Centers  are  the  George  C.  Marshall  European  Center  for  Security 
Studies,  the  Asia-Pacific  Center  for  Security  Studies,  the  Center  for  Hemispheric  Defense 
Studies,  the  Near  East  South  Asia  Center  for  Security  Studies,  and  the  Africa  Center  for 
Security  Studies. 
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ten  focus  areas  for  all  WIF  efforts:  defense  policy  and  strategy;  human 
resource  management;  democratic  control  of  armed  forces;  defense 
planning,  budgeting,  and  resource  management;  logistics;  conceptual, 
planning,  and  operational  aspects  of  peacekeeping;  professional  defense 
and  military  education;  education,  training,  and  doctrine;  emergency 
planning/consequence  management;  and  border  security  and  control. 
WIF  is  not  a  program  per  se  but  rather  an  initiative,  as  discussed  above. 
Therefore,  it  should  be  assessed  in  terms  of  the  numerous  programs 
that  it  supports.10 

The  National  Guard  State  Partnership  Program  provides  unique 
capacity-building  capabilities  to  combatant  commanders  and  U.S. 
ambassadors  through  the  mechanism  of  58  comprehensive  partner¬ 
ships  between  U.S.  states  and  partner  nations.  State  National  Guard 
forces  and  their  partners  participate  in  a  broad  range  of  strategic  secu¬ 
rity  cooperation  activities  on  a  variety  of  topics,  including  homeland 
defense  and  security;  disaster  response  and  mitigation;  consequence 
and  crisis  management;  interagency  cooperation;  border,  port,  and  avi¬ 
ation  security;  combat  medical  internships;  and  bilateral  familiariza¬ 
tion  events. 

The  Global  Train  and  Equip  Program,  sometimes  referred  to  as 
“1206”  because  that  is  the  section  of  law  which  authorized  it,  enables 
DoD  to  conduct  capacity-building  programs  with  foreign  military 
partners.  These  programs  are  aimed  at  improving  the  partner’s  ability 
to  conduct  counterterrorist  operations  or  to  contribute  to  military  and 
stability  operations  in  areas  in  which  the  U.S.  armed  forces  are  also  a 
participant. 

The  Regional  Centers  are  designed  to  build  and  sustain  an 
empowered  international  network  of  current  and  future  security  lead¬ 
ers  who  share  common  values  and  perspectives,  strive  to  increase  their 
national  capacity  to  meet  internal  security  needs  while  contributing 
to  the  security  of  others,  and  promote  greater  international  coopera¬ 
tion.  The  centers  focus  on  network-building  through  resident  execu¬ 
tive  education  and  in-region  conferences,  seminars,  and  workshops. 


10  For  example,  the  Civil  Military  Emergency  Preparedness  Program  and  the  Partnership 
for  Peace  Information  Management  System  program. 
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Focusing  on  regional  audiences  in  a  global  context,  they  widen  per¬ 
spectives  and  enhance  critical  thinking.  Because  the  Regional  Centers 
are  funded  by  several  programs,  assessing  them  at  the  program  level 
must  be  augmented  by  a  higher-level  assessment,  both  individually  and 
across  centers. 

The  Regional  Defense  Combating  Terrorism  Fellowship  Program 
(CTFP)  is  a  security  cooperation  tool  that  provides  education  to  inter¬ 
national  security  personnel  as  part  of  the  global  effort  to  combat  terror¬ 
ism  while  reducing  stress  on  U.S.  forces.  To  achieve  these  goals,  CTFP 
provides  the  geographic  combatant  commanders  with  resources  to 
foster  regional  cooperation  and  to  professionalize  and  expand  foreign 
capabilities  for  combating  terrorism.  The  program  provides  education 
focused  on  combating  terrorism  to  key  senior  and  mid-level  military 
officials,  ministry  of  defense  civilians,  and  other  foreign  government 
security  officials. 

Finally,  the  Overseas  Humanitarian,  Disaster  and  Civic  Aid 
(OHDACA)  Program  enables  DoD  to  assist  countries  by  respond¬ 
ing  to  disasters  in  which  lives  are  at  risk.  OHDACA  authorizes  DoD 
to  provide  services  and  supplies,  logistical  support,  search  and  rescue 
support,  medical  evacuation,  and  refugee  assistance.  The  goals  of  the 
program  include  supporting  U.S.  foreign  policy  and  national  secu¬ 
rity  goals  of  regional  stability,  promotion  of  democracy,  and  economic 
development. 


Organization  of  the  Monograph 

After  this  introduction,  Chapter  Two  provides  an  overview  of  the  May 
2008  assessment  workshop  and  identifies  key  themes  and  challenges 
about  security  cooperation  assessments  by  program  in  order  to  under¬ 
score  the  need  for  the  suggested  assessment  framework. 

Chapter  Three  provides  a  descriptive  overview  of  the  key  elements 
of  a  program-focused  assessment: 

•  strategic  guidance 

•  programs 
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•  stakeholders 

•  authorities  (i.e.,  directives  and  instructions) 

•  the  hve  “levels  of  assessment”  that  are  linked  with  a  discussion  of 
assessment  indicators  (i.e.,  inputs,  outputs,  and  outcomes). 

Chapter  Four  develops  and  illustrates  the  program  assessment 
framework  introduced  in  Chapter  Three.  Chapter  Five  presents  the 
study  team’s  overall  conclusions  and  recommendations.  Appendix 
A  provides  an  in-depth  descriptive  overview  of  the  six  OUSD/P- 
managed  programs  listed  above,  and  Appendix  B  gives  an  overview  of 
the  key  points  from  a  follow-on  assessment  workshop  held  in  October 
2008  that  was  intended  to  implement  one  of  the  key  recommendations 
of  this  study. 


CHAPTER  TWO 


Key  Themes  of  the  Assessment  Workshop 


As  part  of  this  project,  the  RAND  study  team  was  asked  by  the 
OUSD/P  Partnership  Strategy  office  to  design  and  conduct  a  one-week 
assessment  workshop  in  May  2008  for  stakeholders  in  OSD  programs 
for  building  partner  capacity.  The  workshop  drew  about  60  participants 
from  OSD,  the  Defense  Security  Cooperation  Agency,  the  COCOMs, 
services  and  their  component  commands,  and  the  State  Department. 
The  goal  of  the  workshop  was  to  design  an  assessment  framework  for 
the  six  OUSD/P-managed  BPC  programs  described  in  Chapter  One.1 
The  framework  is  intended  to  help  OUSD/P  assess  the  effectiveness  of 
the  programs  in  contributing  to  the  security  cooperation  focus  areas 
contained  in  the  Guidance  for  Employment  of  the  Force  and  to  the 
COCOM  theater  campaign  plans. 

The  workshop  was  organized  so  participants  would  first  address 
broader  security  cooperation  goals  and  then  examine  more  detailed 
assessments  and  their  execution.  The  workshop  was  designed  as 
follows: 

•  Day  1.  Develop  general  agreement  on  the  goals  of  security  coop¬ 
eration  and  types  of  BPC  activities  that  work  toward  those  goals. 

•  Day  2.  Identify  a  handful  of  output  and  outcome  indicators  for 
types  of  security  cooperation  activities,  such  as  short-term  con¬ 
tacts,  exercises  and  field  training,  classroom  education,  equip¬ 
ment  transfer,  and  humanitarian  assistance. 


1  Brief  descriptions  of  these  programs  are  included  in  Appendix  A. 
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•  Day  3.  Set  goals  and  objectives  for  OUSD/P  BPC  programs. 

•  Day  4.  Move  from  ideas  to  implementation  by  determining 
what  is  realistic,  how  assessments  could  be  conducted,  and  how 
much  they  might  cost  in  terms  of  level  of  effort  (i.e.,  money  and 
manpower). 

•  Day  5.  Discuss  ideas  for  next  steps. 

The  study  team  intended  that  workshop  participants  first  gener¬ 
ate  agreement  on  the  broad  goals  and  objectives  of  security  cooperation 
activities  in  general.  Therefore,  in  the  first  two  days  of  the  workshop 
participants  discussed  program  assessment  for  security  cooperation  pro¬ 
grams  before  moving  into  small  group  sessions  on  specific  programs. 
The  main  idea  behind  this  strategy  was  to  determine  common  chal¬ 
lenges,  tasks,  lessons,  and  best  practices  across  programs.  This  strat¬ 
egy  was  perhaps  most  helpful  for  the  participants  from  the  COCOMs, 
who  seldom  think  in  terms  of  individual  programs  but  rather  focus  on 
desired  mid-  and  long-term  effects  or  desired  end-states  in  their  respec¬ 
tive  theaters. 

The  six  BPC  programs  covered  in  the  workshop  were  grouped 
into  three  categories  based  on  their  predominant  activities: 

•  Contact  activities,  which  consist  largely  of  meetings,  conferences, 
and  other  information  exchanges  usually  involving  key  officials, 
action  officers,  and  subject  matter  experts.  Included  programs 
were  Warsaw  Initiative  Fund  and  National  Guard  State  Partner¬ 
ship  Program. 

•  Train-and-equip  and  project-based  activities  include  the  1206  Pro¬ 
gram,  which  transfers  equipment  and  provides  training  to  foreign 
militaries,  and  the  Overseas  Ffumanitarian,  Disaster  and  Civic 
Aid  program,  which  both  trains  and  equips  civilian  response 
agencies.  In  the  case  of  the  1206  Program,  the  goal  is  to  help 
partner  militaries  build  the  capability  to  contribute  to  counter- 
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terrorism  operations  and/or  stability  operations  that  also  include 
U.S.  forces.2 

•  Classroom  education  activities,  which  have  long  played  a  pivotal 
role  in  providing  opportunities  for  foreign  officers  to  attend  U.S. 
military  schools  such  as  the  U.S.  Army  Command  and  General 
Staff  College,  or  counterterrorism  courses  at  the  Regional  Cen¬ 
ters.  Included  programs  were  the  Combating  Terrorism  Fellow¬ 
ship  Program  and  the  Regional  Centers. 

Although  most  programs  consist  of  all  categories  of  activities, 
they  typically  emphasize  one  category  over  the  others.  For  example, 
the  Center  for  Hemispheric  Defense  Studies  has  worked  closely  with 
several  Latin  American  governments  to  develop  national-level  strat¬ 
egy  documents,  and  most  of  the  Regional  Centers  conduct  numerous 
conferences  in  their  areas  on  security  and  defense  topics.  In  fact,  all 
the  Regional  Centers  conduct  alumni  activities  outside  the  classroom 
experience,  which  could  easily  be  categorized  as  contact  activities.  Yet 
education  remains  their  primary  activity. 

Defining  categories  of  activities  made  it  easier  for  the  workshop 
participants  to  see  similarities  between  programs  that  on  the  surface 
appear  to  have  little  in  common.  For  example,  the  types  of  outreach 
activities  conducted  by  the  Regional  Centers  are  nearly  identical  to 
the  subject  matter  exchanges  supported  by  WIF  or  conducted  by  the 
National  Guard  SPP.  WIF  funds  U.S.  civilian  defense  officials  to  help 
foreign  governments  build  civilian  oversight  mechanisms  for  their 
defense  ministries,  much  like  the  Center  for  Hemispheric  Defense 
Studies  work  in  Latin  America  discussed  above.  Likewise,  the  National 
Guard  holds  conferences  on  a  range  of  topics  for  its  partner  countries, 
including  security  and  defense  topics  similar  to  ones  funded  and  con¬ 
ducted  by  the  Regional  Centers. 

Once  the  commonalities  between  programs  were  identified  by  the 
participants,  it  was  easier  to  design  a  comprehensive  assessment  frame- 


2  Nina  Serafino,  “Section  1206  of  the  National  Defense  Authorization  Act  of  FY2006: 
A  Fact  Sheet  on  Department  of  Defense  Authority  to  Train  and  Equip  Foreign  Military 
Forces,”  Congressional  Research  Service  Report  to  Congress,  updated  May  15,  2008. 
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work  to  capture  the  effectiveness  of  these  types  of  activities,  regardless 
of  the  program  involved.  That  framework  is  the  subject  of  the  next 
chapter. 


Main  Themes 

The  lengthy  and  lively  discussions  in  the  workshop  are  summarized 
here  into  the  following  themes:  setting  direction,  designing  assess¬ 
ments,  preparing  for  assessment,  conducting  assessments,  and  explain¬ 
ing  results.  Together,  these  themes  underscore  the  pressing  need  for  a 
program  assessment  framework  for  security  cooperation. 

Setting  Direction 

Workshop  participants  noted  that  setting  specific  direction  for  BPC 
and,  indeed,  for  broader  security  cooperation  programs  is  a  challenge, 
particularly  because  most  programs  support  multiple  end-states  and 
multiple  stakeholders.  For  example,  the  National  Guard  State  Part¬ 
nership  Program  has  the  ability  to  draw  on  a  wide  variety  of  exper¬ 
tise  from  within  the  National  Guard  to  support  the  military  security 
cooperation  priorities  of  each  geographic  COCOM,  U.S.  ambassadors, 
or  the  services.  From  the  perspective  of  building  relationships,  almost 
any  activity  is  justifiable,  so  long  as  it  is  seen  as  improving  the  existing 
relationship.  Yet  BPC  programs  are  also  designed  to  support  the  related 
longer-term  goal  of  improving  the  capacity  of  partners  through  build¬ 
ing  and  maintaining  relationships,  building  partner  capabilities,  and 
enhancing  interoperability  with  U.S.  forces.  This  gap  between  specific 
operational  short-term  goals  and  the  longer-term  BPC  goals  creates  a 
challenge  for  planners  and  program  managers  in  assessing  program 
effectiveness.  The  challenge  is  reflected  in  the  GEF  itself,  which  lays 
out  implicit  BPC  tasks  through  “security  cooperation  focus  areas”  but 
lacks  specific  linkages  between  contingency  planning  and  program 
activities. 

To  highlight  the  long-term  nature  of  BPC  programs,  some  par¬ 
ticipants  recommended  that  OUSD/P  consider  establishing  goals  in 
cooperation  with  other  U.S.  government  agencies,  in  order  to  build 
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partner  capacity  for  the  entire  partner  country,  not  just  its  military. 
For  example,  training  and  equipping  foreign  militaries  can  be  a  waste 
of  effort  if  the  foreign  government  itself  is  corrupt  or  too  ineffective  to 
maintain  the  equipment  it  has  been  given.  In  that  circumstance,  the 
first  priority  is  not  building  the  capacity  of  the  military  partner  but 
rather  employing  State  Department  and  U.S.  Agency  for  International 
Development  (USAID)  programs  to  improve  the  partner’s  basic  gov¬ 
erning  and  economic  capacity. 

Also,  since  most  programs  support  multiple  end-states  and  there¬ 
fore  multiple  stakeholders,  it  is  necessary  to  identify  the  customer 
for  each  assessment.  Workshop  participants  felt  it  is  important  that 
policy  be  developed  to  clarify  the  roles  and  missions  for  decisionmak¬ 
ing  authority,  resourcing,  data  and  reporting  requirements,  building 
objectives,  and  time  frames.  For  example,  the  National  Guard  State 
Partnership  Program  operates  in  several  theaters  under  slightly  differ¬ 
ent  operating  conditions,  depending  on  the  country.  Each  theater  has 
different  structures  and  requirements.  So  in  this  case,  should  the  SPP 
address  its  program  assessment  to  the  National  Guard,  the  ambassador, 
the  COCOM,  or  OUSD/P?  The  assessment  framework  established 
in  Chapter  Three  helps  to  clarify  the  roles  and  missions  of  varying 
stakeholders. 

The  participants  emphasized  building  security  cooperation  goals 
that  can  stand  the  test  of  time.  Enduring  goals  and  more  specific  objec¬ 
tives  are  essential  to  enable  assessments  over  time. 

Finally,  to  compensate  for  the  GEF’s  lack  of  guidance  in  the  area 
of  security  cooperation  program  assessment,  they  suggested  that  it  may 
help  to  standardize  the  assessment  process — for  example,  by  making 
linkages  explicit  and  assessment  terminology  uniform. 

Designing  Assessments 

The  workshop  participants  discussed  at  length  both  challenges  and 
best  practices  in  designing  program  assessments.  Over  the  course  of 
the  week,  discussion  focused  on  the  types  of  assessment  needed,  how  to 
build  measurable  objectives  into  plans,  balancing  objective  and  subjec¬ 
tive  assessments,  building  output  and  outcome  indicators  for  different 
contexts,  measuring  the  sustainability  of  partner  capabilities  over  time, 
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and  fixing  inconsistent  language  and  terminology.  The  following  sub¬ 
sections  describe  these  discussions  in  greater  detail. 

Two  types  of  assessment:  efficiency  and  effectiveness.  One  type 
of  assessment  is  concerned  primarily  with  efficiency,  or  how  well  pro¬ 
grams  fill  requests  and  provide  goods  and  services  to  the  COCOMs  in 
order  to  build  the  desired  capacity  of  partner  militaries.  Are  they  pro¬ 
viding  what  is  required?  A  second  type  of  assessment  focuses  on  effec¬ 
tiveness,  or  whether  or  not  a  theater  commander  has  reached  a  desired 
outcome  or  effect  in  terms  of  building  partner  capacity.  To  measure 
effectiveness,  security  cooperation  planners  need  to  establish  a  baseline 
for  the  partner’s  military  capabilities  from  which  to  judge  progress  or 
the  lack  thereof.  However,  at  present  most  programs  do  not  focus  on 
assessing  baseline  capabilities;  that  is  usually  the  role  of  the  COCOM. 
In  any  event,  both  types  of  assessment  are  necessary — the  efficiency  of 
the  program  managers  themselves  in  providing  goods  and  services  to 
the  COCOMs,  and  the  ultimate  effectiveness  of  their  activities  mea¬ 
sured  against  regional  and  U.S.  security  goals. 

Building  measurable  objectives  into  plans.  Although  OUSD/P 
BPC  programs  are  managed  by  OUSD/P  and  therefore  are  assessed 
against  OUSD/P  security  cooperation  goals,  the  programs  are  now 
required  to  support  COCOM  theater  campaign  plans  as  well.3  In  the 
past,  programs  did  not  make  an  explicit  connection  with  the  theater 
security  cooperation  plans  and  instead  focused  on  assessing  their  prog¬ 
ress  against  OUSD/P  security  cooperation  goals  in  general.  However, 
now  that  the  GEF  calls  for  program  assessment  against  theater  cam¬ 
paign  plans,  it  is  important  to  design  a  similar  assessment  process  across 
COCOMs  so  that  the  efficiency  and  effectiveness  of  programs  can  be 
assessed.  Until  now,  specific  measurable  program  objectives  have  not 
been  included  in  the  COCOM  theater  campaign  plans,  and  it  remains 
unclear  how  theater  commanders  will  articulate  how  they  intend  to  use 
BPC  programs  to  accomplish  their  broader  security  cooperation  goals. 

In  addition,  each  COCOM  has  slightly  different  objectives.  For 
example,  SOUTHCOM  (U.S.  Southern  Command)  may  focus  pri- 


3  As  discussed  above,  most  BPC  programs  support  a  variety  of  long-term  goals;  these 
include  both  OSD  and  COCOM  goals. 
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marily  on  building  the  capacity  of  partners  to  conduct  operations  at 
the  lower  end  of  the  operational  spectrum,  such  as  consequence  man¬ 
agement,  disaster  relief,  and  peacekeeping;  the  U.S.  Central  Command 
(CENTCOM)  may  focus  primarily  on  building  partner  capacity  to 
conduct  counterterrorism  operations.4  As  a  result,  although  the  vari¬ 
ous  COCOMs  design  their  plans  in  a  similar  fashion,  their  priorities 
will  necessarily  differ.  And  programs  now  must  tailor  their  assessments 
to  measure  their  effectiveness  against  the  objectives  of  six  geographic 
COCOM  plans. 

To  provide  some  consistency  for  program  assessment,  the  work¬ 
shop  participants  suggested  that  COCOM  planners  build  more  mea¬ 
surable  objectives  into  their  plans.  In  the  absence  of  measurable  objec¬ 
tives  at  the  COCOM  level,  some  participants  suggested  that  programs 
build  subobjectives  linked  to  higher-level  OUSD/P  objectives,  such  as 
the  Security  Cooperation  Focus  Areas,  that  are  generally  consistent 
over  a  longer  period  of  time.  In  this  way,  programs  could  be  focused 
on  strategic  goals  and  could  build  a  history  of  what  had  been  asked  for 
and  what  was  provided  at  the  regional  level  in  relation  to  high-level 
goals  and  objectives. 

Balancing  objective  and  subjective  assessments.  As  mentioned 
earlier,  past  assessments  have  been  mostly  subjective  and  were  done  by 
program  managers  or  at  the  COCOMs.  For  example,  program  manag¬ 
ers  are  asked  to  rate  their  events  and  their  ability  to  support  numerous 
security  cooperation  objectives,  and  the  COCOMs  are  asked  to  pro¬ 
vide  input  on  the  broad  effectiveness  of  all  the  programs  operating  in 
their  theater. 

Workshop  participants,  however,  generally  regarded  this  system  of 
subjective  self-assessment  as  no  longer  sufficient.  That  said,  the  group 
also  opposed  assessing  security  cooperation  activities  and  programs 
on  purely  objective  output  measures — such  as  numbers  of  students  in 
courses,  amount  of  money  spent  on  exercises,  or  number  of  visits  per 
country — since  many  of  the  more  positive  effects  of  cooperation  with 
foreign  countries  are  based  on  establishing  long-term  relationships  and 


4  This  is  not  to  say  that  COCOMs  do  not  build  partner  capacity  across  the  entire  spec¬ 
trum,  just  that  each  COCOM  must  tailor  its  plans  to  what  is  required  in  its  area. 
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trust,  something  that  is  inherently  difficult  to  quantify.  Therefore,  the 
workshop  consensus  was  that  program  assessment  design  should  focus 
on  outcomes-based  assessments  that  take  into  account  both  subjective 
and  objective  data  in  order  to  measure  both  the  efficiency  with  which 
the  programs  operate  (doing  things  right)  and  the  effectiveness  of  pro¬ 
grams  over  time  (doing  the  right  things). 

Putting  output  and  outcome  indicators  in  context.  Participants 
remarked  that  assessing  whether  or  not  a  program  has  had  influence  or 
has  helped  build  capacity  requires  programs  to  build  indicators  that  are 
easy  to  use  and  understand.  For  example,  how  will  the  program  spe¬ 
cifically  develop  influence  with  a  particular  country  or  on  a  particular 
issue?  What  specific  capability  will  the  program  help  to  build?  At  the 
same  time,  indicators  should  take  into  account  that  what  works  in  one 
context  might  not  work  in  another.  For  example,  classroom  education 
may  improve  the  ability  of  senior  military  staff  to  command  subordi¬ 
nate  units  in  one  context,  but  field  training  may  be  more  appropriate 
for  company-level  staff  and  below.  Programs  can  have  a  global  focus, 
but  effectiveness  may  need  to  be  assessed  against  goals  that  are  specific 
to  the  context.  In  the  end,  it  is  important  to  know  whether  programs 
have  had  influence  and  have  helped  build  capabilities  and  capacity 
needed  by  the  partners  as  well  as  COCOMs. 

Measuring  the  sustainability  of  partner  capabilities  over  time. 
Many  participants  felt  that  current  security  cooperation  assessments 
fail  to  measure  whether  partners  can  sustain  capabilities  over  time, 
even  though  this  is  perhaps  the  most  important  element  of  building 
partner  capacity.  Too  often  programs  are  assessed  based  on  whether  or 
not  they  provided  the  training,  equipment,  or  opportunities  requested 
by  the  COCOM  or  partner  nation  (an  output  metric),  and  this  is 
where  the  assessment  ends.  It  is  then  left  to  others  to  decide  whether  or 
not  the  partner  actually  increased  its  capability.  For  example,  a  part¬ 
ner’s  military  personnel  may  receive  English  language  training,  but  if 
those  trained  personnel  with  English  skills  leave  the  military,  the  new 
capability  is  lost.  Under  current  assessment  standards,  which  focus  on 
efficiency,  the  English  training  program  would  be  considered  a  suc¬ 
cess  because  the  training  was  provided.  However,  if  the  program  were 
measured  by  the  effectiveness  of  the  training  for  improving  the  part- 
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ner  military’s  level  of  English  capability,  especially  if  the  assessment  is 
based  on  how  those  military  personnel  used  their  new  skill  (an  out¬ 
come  objective),  the  program  would  not  be  judged  successful. 

More  generally,  the  ability  to  measure  sustainability  is  hindered 
by  both  the  difficulty  of  creating  capability  baselines  and  the  challenge 
of  synchronizing  myriad  security  cooperation  programs  and  funding 
sources.  Current  BPC  practice  relies  on  stitching  together  multiple 
activities  and  events.  For  example,  the  Georgia  Train  and  Equip  Pro¬ 
gram5  relied  on  many  different  programs  to  provide  everything  from 
English  training  for  Georgian  troops  and  help  with  defense  budgeting, 
to  basic  equipment  and  military  advisors  for  training  the  troops.  There 
is  little  disagreement  that  this  program  helped  to  create  capability  in 
the  Georgian  military.  However,  when  it  comes  to  program  assessment, 
it  is  difficult  to  evaluate  the  contributions  of  one  program  relative  to 
another. 

According  to  some  participants,  one  way  to  approach  this  chal¬ 
lenge  is  for  program  managers  to  develop  measures  for  their  own  pro¬ 
grams  and  activities  that  reflect  a  longer-term  view  of  effectiveness. 
However,  creating  this  long-term  view  at  the  program  level  again  comes 
back  to  the  challenge  of  establishing  a  proper  baseline,  or  starting 
point,  from  which  to  assess.  Workshop  participants  agreed  that  what 
is  most  important  to  the  program  should  drive  the  choice  of  that  base¬ 
line.  For  example,  education  programs  should  be  judged  on  what  they 
have  accomplished  over  the  history  of  their  program.  After  all,  many 
partner  nations  send  their  most  capable  young  officers  to  school  in  the 
United  States,  but  the  effect  of  this  education  may  not  be  known  until 
those  officers  rise  to  positions  of  power  within  their  military  depart¬ 
ments.  Education  programs  could  track  where  their  previous  graduates 
are  now  assigned  as  a  way  to  assess  the  effectiveness  of  their  courses  in 
the  past.  Their  baseline  might  be  the  number  of  senior  officers  with 
U.S.  education  experience  when  the  programs  began.  In  another  exam¬ 
ple,  train-and-equip  programs  might  start  with  a  current  assessment  of 


5  The  Georgia  Train  and  Equip  Program  took  place  in  Georgia  from  May  2002  to  April 
2004  and  cost  approximately  $64  million.  Training  was  provided  to  2,600  individuals  from 
the  Ministries  of  Defense  and  Interior. 
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partner  capability,  such  as  individual  and  collective  military  skills,  and 
develop  indicators,  or  benchmarks,  that  look  forward  to  the  time  when 
the  partner  gains  the  new  capability. 

Fixing  inconsistent  language  and  terminology.  Everyone  agreed 
that  consistent  assessment  terminology  is  needed  across  DoD  and  pref¬ 
erably  across  other  non-DoD  departments  and  agencies  as  well.  The 
terms  used  by  planners  and  assessors  are  not  all  that  different,  but  the 
differences  that  do  exist  continue  to  make  it  difficult  to  develop  stan¬ 
dard  assessment  frameworks  across  the  security  cooperation  commu¬ 
nity.  For  example,  OUSD/P  uses  the  term  “measure  of  progress”  in  the 
GEF,  the  COCOMs  measure  “effects,”  and  services  and  agencies  are 
more  accustomed  to  measures  of  “performance”  and  “effectiveness.” 

Standardizing  the  process  by  building  a  common  vocabulary  and 
a  uniform  data  collection  process  would  add  to  the  assessment’s  utility 
because  it  would  enhance  transparency  and  boost  confidence  in  the 
planning  process.  As  it  is  now,  the  reliance  on  self-assessments  by  pro¬ 
grams  and  superficial  assessments  by  the  COCOMs  leaves  the  result 
open  to  criticism  and  makes  resourcing  decisions  difficult.  However, 
if  everyone — program  managers  and  planners  alike — used  a  similar 
assessment  framework  and  methodology,  it  would  allow  OUSD/P  and 
others  to  assess  the  performance  and  effectiveness  of  programs  across 
COCOMs. 

Preparing  for  Assessment 

Preparing  stakeholders,  fn  many  ways,  preparing  stakeholders 
for  assessment  is  just  as  important  as  designing  the  assessment  in  the 
first  place.  All  the  workshop  participants  acknowledged  that  program 
managers  cannot  assess  the  effectiveness  of  their  programs  without  the 
help  and  collaboration  of  many  different  stakeholders.  Participants  felt 
that  it  is  sometimes  difficult  to  convince  stakeholders,  including  those 
who  own  useful  data,  that  assessment  is  worthwhile.  Some  participants 
suggested  developing  an  education  process  for  program  managers,  pro¬ 
gram  recipients,  and  other  stakeholders,  perhaps  through  assessment 
training  and  seminars,  to  better  prepare  them  for  the  assessments.  A 
considerable  effort  is  needed  to  ensure  everyone  is  on  board  and  col¬ 
lecting  the  right  data  for  the  right  purposes. 
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Hiring  personnel.  Participants  also  lamented  the  lack  of  person¬ 
nel  with  appropriate  assessment  backgrounds  (for  instance,  operations 
research  or  program  management  training),  especially  in  the  military; 
once  officials  learn  the  intricacies  of  a  program  and  the  assessment  pro¬ 
cess,  their  tours  of  duty  are  over  and  their  experience  and  knowledge 
are  lost  to  the  program  when  they  move  on.  Participants  said  it  would 
be  useful  to  have  a  handful  of  experienced  assessment  professionals 
who  could  help  programs  and  COCOMs  alike,  especially  in  learn¬ 
ing  such  assessment  basics  as  setting  baselines  and  developing  practi¬ 
cal  output  and  outcome  indicators,  as  well  as  understanding  the  wider 
security  cooperation  community,  other  stakeholder  roles,  and  associ¬ 
ated  program  regulations  and  authorities. 

Conducting  Assessments 

Data  standardization  and  aggregation.  It  is  still  a  challenge  to 
ensure  that  assessment  data  are  not  lost  or  altered  during  aggregation. 
Currently,  every  stakeholder — each  COCOM,  for  instance — has  its 
own  system.  In  addition,  programs  also  have  developed  their  own 
data  collection  mechanisms  and  standards.  This  makes  it  impossible 
to  know  if  one  program  is  more  or  less  effective  than  another,  let  alone 
better  or  worse  in  different  regions  of  the  world. 

Second,  even  if  there  were  a  standardized  data  collection  system, 
participants  believe  that  the  meaning  behind  the  data  often  would  still 
be  lost  as  information  is  processed  and  aggregated  to  higher  levels.  To 
ensure  that  the  aggregated  data  do  not  lose  their  meaning,  some  par¬ 
ticipants  suggested  that  the  system  should  take  tactical-level  data,  such 
as  personnel  exchanges  or  individual  classes  for  foreign  military  per¬ 
sonnel,  and  consider  their  strategic-level  implication.  For  example,  how 
does  the  class  or  personnel  exchange  enhance  partner  capacity?  While 
it  is  inherently  difficult  to  link  individual  events  to  strategic  goals, 
as  was  mentioned  above,  some  participants  suggested  that  OUSD/P 
could  assign  such  identifying  “tags”  to  the  data  in  different  systems 
that  would  link  specific  activities  or  events  to  certain  security  coopera¬ 
tion  goals. 

Data  collection.  Although  programs  can  use  a  variety  of  data 
sources — the  partner  nations  themselves,  U.S.  embassies,  nongovern- 
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mental  agencies,  and  others  in  the  held — workshop  participants  rec¬ 
ognized  that  data  collection  is  also  a  challenge.  Some  participants  sug¬ 
gested  exploring  external  indicators,  such  as  socioeconomic  indicators 
or  commissioned  surveys,  and  developing  multiple  data  sources  beyond 
program  managers  and  U.S.  personnel. 

Assessment  resources.  Others  suggested  sending  teams  into  the 
held  to  assess  key  effects.  They  also  suggested  perhaps  setting  up  a  new 
office  that  would  conduct  program  assessments  and  offer  advanced 
assessment  training  to  program  managers  and  security  cooperation 
planners  in  the  held.  Based  on  their  experience  with  other  U.S.  govern¬ 
ment  programs,  some  participants  suggested  that  a  “tax”  (perhaps  1-3 
percent)  could  be  levied  on  programs  to  cover  the  costs  of  the  shared 
assessment  resource. 

Classification  of  data.  Finally,  everyone  agreed  that  the  classifica¬ 
tion  level  of  assessment  data  is  a  difficult  challenge.  Knowing  if  a  pro¬ 
gram  is  really  effective  requires  understanding  the  context  of  the  data, 
which  involves  specihc  information  on  the  capabilities  and  capacity  of 
foreign  countries.  Participants  acknowledged  that  classihcation  is  inev¬ 
itable  but  worried  that  personnel  in  the  held  need  access  to  the  infor¬ 
mation  and  also  require  feedback  from  higher  headquarters  on  what 
is  effective  and  what  is  not.  Some  participants  stated  that  as  policy 
changes,  it  will  be  easier  to  handle  both  unclassihed  and  classihed  data 
on  the  same  systems.  Until  then,  however,  limited  access  to  classihed 
assessment  data  will  make  feedback  to  the  held  difficult. 

Explaining  Assessments 

Connect  DoD  to  U.S.  foreign  policy  goals.  DoD  may  hnd  more 
support  for  BPC  programs  if  it  can  explain  how  its  efforts  are  tied  to 
larger  U.S.  foreign  policy  goals.  To  this  end,  participants  stated  that  it 
is  essential  for  DoD  to  understand  how  security  cooperation  programs 
complement  (or  impede)  other  efforts  of  the  military  and  other  govern¬ 
ment  agencies,  such  as  the  State  Department  or  USAID.  The  military 
recognizes  that  building  partner  capacity  for  security  is  important  but 
that  doing  so  is  more  effective  and  sustainable  when  it  is  combined  with 
other  U.S.  government  efforts  in  economic  and  social  development. 
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Build  a  transparent  assessment  process.  All  workshop  partici¬ 
pants  agreed  that  it  is  vital  for  DoD  to  better  explain  the  assessment 
process  to  assessment  audiences,  such  as  Congress,  skeptical  customers, 
and  other  key  stakeholders.  The  best  way  for  DoD  to  improve  trans¬ 
parency  with  others  is  to  build  a  common  assessment  framework  and 
collection  system  that  incorporates  feedback  mechanisms  to  the  field. 
Participants  acknowledged  that  this  effort  will  challenge  existing  orga¬ 
nizational  cultures  that  have  been  operating  (in  some  cases  for  years) 
without  effective  assessment  mechanisms. 


Summary 

The  challenges  for  BPC  and  broader  security  cooperation  program 
assessment  are  significant,  but  participants  in  the  week-long  assessment 
workshop  found  substantial  agreement  during  the  discussion  of  oppor¬ 
tunities  for  setting  direction  as  well  as  designing,  preparing  for,  con¬ 
ducting,  and  explaining  assessment.  In  particular,  they  found  many 
similarities  across  their  programs  and  perspectives.  By  focusing  on  cat¬ 
egories  of  activities — such  as  military-to-military  contact,  training  and 
equipping,  and  education — the  program  managers,  COCOM  staff, 
and  service  representatives  found  areas  of  agreement  on  the  major  tasks 
and  fundamental  elements  needed  to  assemble  an  improved,  consistent 
assessment  process. 

In  Chapter  Three,  we  expand  the  discussion  and  begin  to  outline 
specific  elements  of  a  framework  for  assessing  the  effectiveness  of  OSD 
Title  10  BPC  programs. 


CHAPTER  THREE 


Key  Elements  of  the  Assessment  Program 
Framework 


This  chapter  provides  a  descriptive  overview  of  the  five  key  elements  of 
a  program-focused  assessment  framework: 

•  strategic  guidance 

•  programs 

•  stakeholders 

•  authorities  (including  directives  and  instructions) 

•  “levels  of  assessment”  that  are  linked  with  a  discussion  of  assess¬ 
ment  indicators  (inputs,  outputs,  and  outcomes). 

The  discussion  of  each  element  in  this  chapter  sets  the  stage  for  the 
following  chapter,  which  concerns  integrating  and  implementing  the 
elements  of  the  framework  in  order  to  assess  security  cooperation 
programs. 


Strategic  Guidance 

At  the  highest  levels,  the  National  Security  Strategy,  the  National 
Defense  Strategy,  the  National  Military  Strategy,  the  OSD  Guidance 
for  Employment  of  the  Force,  and  the  COCOM  theater  campaign 
plans  form  the  basis  for  strategy-driven  security  cooperation  and  are 
key  documents  in  selecting  priority  partners  of  interest  to  the  United 
States.  The  military  departments  implement  the  OSD  guidance  and 
support  the  COCOM  theater  campaign  plans  by  developing  their  own 


31 


32  A  Framework  to  Assess  Programs  for  Building  Partnerships 


campaign  support  plans.  Figure  3.1  provides  an  overview  of  these  guid¬ 
ance  documents  and  shows  how  they  relate  to  one  another. 

The  National  Security  Strategy,  which  provides  top-level  strate¬ 
gic  guidance  to  DoD  and  other  departments,  emphasized  in  March 
2006  that  the  United  States  must  gain  the  support  and  active  coopera¬ 
tion  of  friends  and  allies.  In  this  spirit,  the  National  Defense  Strategy, 
the  DoD’s  internal  strategic  guidance  document,  addresses  the  need  to 
strengthen  alliances  and  partnerships.1  It  emphasizes  that  the  United 
States  currently  does  not  have  the  capacity  to  address  all  global  security 
challenges  without  assistance  and  will  require  the  support  of  the  inter¬ 
national  community  for  the  foreseeable  future.  The  Chairman  of  the 
Joint  Chiefs  of  Staff  develops  the  National  Military  Strategy  to  imple- 

Figure  3.1 
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1  Office  of  the  Secretary  of  Defense,  National  Defense  Straegy  of  the  United  States,  Washing¬ 
ton,  D.C.,  March  2005,  p.  iv. 
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ment  the  National  Defense  Strategy  and  instructs  the  military  depart¬ 
ments  to  enable  “multinational  partners  through  security  cooperation 
and  other  engagement  activities.”2 

The  key  point  of  these  strategic  documents  is  the  importance  of 
developing  capabilities  and  cooperating  with  partner  militaries  to  meet 
U.S.  strategic  goals.  Based  on  the  strategic  guidance,  OSD  produces 
the  GEF,  which  COCOMs  use  as  the  basis  for  developing  their  theater 
campaign  plans. 

The  GEF  outlines  eight  security  cooperation  areas: 

•  operational  access  and  global  freedom  of  action 

•  operational  capacity  and  capability 

•  interoperability 

•  intelligence-  and  information-sharing 

•  assurance  and  regional  confidence-building 

•  security-sector  reform 

•  defense  exports  and  international  collaboration 

•  national  and  multinational  influence. 

These  focus  areas  are  intended  to  help  the  COCOMs,  services,  and 
defense  agencies  focus  their  security  cooperation  efforts  with  partner 
countries. 


Programs 

Security  cooperation  programs  are  included  in  strategic  guidance 
within  the  following  categorization  scheme: 

•  combined/multinational  education 

•  combined/multinational  exercises 

•  combined/multinational  experimentation 

•  combined/multinational  training 

•  counternarcotics  assistance 


2  Chairman,  Joint  Chiefs  of  Staff,  The  National  Military  Strategy  of  the  United  States  of 
America:  A  Strategy  for  Today;  A  Vision  for  Tomorrow,  2004,  p.  8. 
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•  counter/nonproliferation 

•  defense  and  military  contacts 

•  defense  support  to  public  diplomacy 

•  facilities  and  infrastructure  support  projects 

•  humanitarian  assistance 

•  information  sharing/intelligence  cooperation 

•  international  armaments  cooperation 

•  security  assistance 

•  other  programs  and  activities. 

It  has  become  an  accepted  practice  for  major  security  cooperation  orga¬ 
nizations  (e.g.,  OUSD/P,  services,  COCOMs,  DSCA)  to  develop  their 
own  list  of  security  cooperation  categories  or  types  of  activities,  which 
they  periodically  modify.  Although  these  lists  share  many  elements  in 
common,  the  practice  is  likely  to  continue. 

However,  despite  OUSD/P’s  stamp  of  approval  and  a  growing 
consensus  on  the  need  for  a  standardized  list,  the  GEF  categorization 
of  security  cooperation  areas  is  unlikely  to  be  definitive.  One  reason  is 
that  the  current  GEF  list  contains  missions,  such  as  counternarcotics 
and  counter-  and  nonproliferation,  that  cut  across  various  BPC  activi¬ 
ties,  such  as  workshops,  training,  and  exercises.  Moreover,  security 
assistance  is  generally  considered  to  encompass  all  Title  22  (Depart¬ 
ment  of  State-controlled)  security  cooperation  programs,  including 
education,  training,  equipping,  and  other  kinds  of  activities,  not  all  of 
which  are  on  the  GEF  list.  Nor  does  the  GEF  list  include  workshops 
and  conferences,  which  constitute  a  large  share  of  DoD  security  coop¬ 
eration  activities  and  are  functionally  distinct  from  defense  and  mili¬ 
tary  contacts.  For  assessment  purposes,  it  would  be  useful  to  separate 
missions  from  the  categories  of  activities,  eliminate  the  security  assis¬ 
tance  category  since  it  is  duplicative,  and  list  workshops  and  confer¬ 
ences  as  a  separate  category. 

As  mentioned  earlier,  security  cooperation  programs  are  managed 
and  resourced  and  pursue  their  activities  in  different  ways.  They  are 
found  at  many  levels  within  DoD.  There  are  OUSD/P-managed  pro¬ 
grams,  such  as  Global  Train  and  Equip  (i.e.,  1206)  and  the  Warsaw  Ini¬ 
tiative  Fund,  which  provides  assistance  to  Partnership  for  Peace  nations 
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by  building  defense  institudons.  OUSD/P  controls  the  resources  and 
makes  decisions  regarding  rhe  overall  outcome  objectives  for  WIF. 
Other  programs  are  managed  by  the  Joint  Staff,  for  example,  Chair¬ 
man’s  Exercise  Program  and  Joint  Staff  talks  with  partner  countries. 
Still  other  programs,  such  as  the  Joint  Contact  Team  Program,  are 
managed  by  the  COCOMs — in  this  case  by  the  U.S.  European  Com¬ 
mand  (EUCOM).  Combat  support  agencies,  for  example,  rhe  Defense 
Threat  Reduction  Agency,  manage  programs  such  as  the  International 
Counterproliferation  Program.  Finally,  the  military  services  manage 
many  of  their  own  programs — such  as  International  Armaments  Coop¬ 
eration,  the  Military  Personnel  Exchange  Program,  and  staff  talks  with 
partner  countries. 

To  complicate  matters  still  further,  the  stakeholders  managing 
a  program  often  do  not  actually  execute  the  program’s  activities  and 
events.  The  programs  under  OUSD/P’s  Warsaw  Initiative  Fund  are 
executed  by  a  variety  of  stakeholders,  including  the  COCOMs,  the 
services  (specifically  the  Army  and  Air  Force),  and  private-sector  con¬ 
tractors.  The  OUSD/P  Combating  Terrorism  Fellowship  Program  is 
managed  by  OUSD/P  but  executed  by  the  DoD  Regional  Centers  and 
schoolhouses,  such  as  the  National  War  College.  Most,  but  not  all,  of 
these  relationships  are  defined  in  the  security  cooperation  authorities, 
as  discussed  below. 


Stakeholders 

In  broad  usage,  a  stakeholder  in  a  program  is  a  person  or  organization 
that  affects  or  might  be  affected  by  the  actions  of  that  program.  How¬ 
ever,  in  this  monograph  we  use  the  term  stakeholders  to  refer  to  assess¬ 
ment  stakeholders — defined  more  narrowly  as  persons  or  organizations 
within  DoD  that  make  decisions  for  or  about  a  program.  This  narrower 
definition  is  intended  to  keep  assessment  focused  on  decisions.  Ancil¬ 
lary  stakeholders  who  are  curious  about  a  program  or  who  are  down¬ 
stream  and  affected  by  it  but  do  not  make  or  contribute  to  decisions  are 
not  assessment  stakeholders.  They  may  provide  data  that  contribute  to 
assessments  but  should  not  be  defining  assessment  needs. 
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Security  cooperation  stakeholders  can  be  thought  of  as  the  vari¬ 
ous  organizations  that  are  involved  with  overseeing,  planning,  or 
implementing/executing  security  cooperation  programs.  Often,  many 
stakeholders  are  connected  to  a  given  program  in  some  way.  For  exam¬ 
ple,  the  OUSD/P  Warsaw  Initiative  Fund  includes  several  programs, 
such  as  the  Civil  Military  Emergency  Preparedness  Program.  This 
single  WIF  program  is  executed  by  the  U.S.  Army  (i.e.,  Headquarters, 
G-35  Stability  Operations  and  the  Army  Corps  of  Engineers)  and  sev¬ 
eral  contractors.  Depending  on  the  nature  of  the  event  being  executed, 
it  also  involves  partnering  with  DoD  organizations,  such  as  DTRA, 
and  other  U.S.  government  departments,  such  as  the  Department  of 
Homeland  Security.  For  example,  Homeland  Security  officials  might 
be  invited  to  Civil-Military  Emergency  Preparedness  events  focused  on 
cooperation  among  the  various  partner  security  forces  for  homeland 
security  purposes.  The  other  programs  funded  by  WIF — the  Partner¬ 
ship  for  Peace  Information  Management  System,  logistic  exchanges, 
Regional  Airspace  Initiative  studies,  and  Defense  Resource  Manage¬ 
ment  exchanges — all  have  similarly  complicated  combinations  of 
stakeholders. 

Understanding  the  specific  role  of  each  stakeholder — be  it  plan¬ 
ning,  resource  management,  or  implementation — provides  a  clearer 
picture  of  its  appropriate  assessment  responsibilities. 


Authorities 

DoD  stakeholders  derive  their  individual  authorities  for  conducting 
security  cooperation  programs  from  the  U.S.  Code,  DoD  directives, 
and  instructions.  Title  10  of  the  U.S.  Code  (U.S.C.)  is  the  basic  author¬ 
ity  for  DoD  activities;  it  also  serves  as  the  primary  authority  for  many 
DoD  security  cooperation  programs. 

In  contrast,  Title  22  of  the  U.S.  Code  provides  the  basic  author¬ 
ity  for  foreign  assistance,  including  security  assistance.  As  mentioned 
earlier,  Title  22  is  distinctive  because,  although  policy  and  resources 
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are  largely  the  responsibility  of  the  State  Department,  security  assis¬ 
tance  is  administered  by  DoD.  Title  22  security  assistance  programs 
and  funding  sources  include  Foreign  Military  Sales,  Foreign  Military 
Financing,  International  Military  Education  and  Training,  Direct 
Commercial  Sales,  presidential  drawdown,  excess  defense  articles,  and 
equipment  leases. 

Various  laws,  directives,  and  instructions  govern  the  execution  of 
the  security  cooperation  programs.  But  not  all  programs  have  associ¬ 
ated  directions  or  instructions.  For  example,  the  WIF  is  governed  by 
10  U.S.C.  168,  1051,  and  2010,  but  does  not  have  an  associated  direc¬ 
tive.  The  1206  Global  Train  and  Equip  Program  is  another  case  in 
point.  Its  authority  is  derived  directly  from  Public  Law  (PL)  109-163, 
Section  1206,  and  is  an  exception  to  the  existing  DoD  authorities. 
The  Joint  Staff  Chairman’s  exercises  are  governed  by  10  U.S.C.  153 
and  the  Joint  Training  Manual for  the  Armed  Forces  of  the  United  States 
(CJCSM  3500. 03A).  The  services  also  have  varying  degrees  of  direc¬ 
tives  and  instructions  attached  to  their  programs.  For  example,  the  Air 
Force’s  instruction  for  the  Military  Personnel  Exchange  Program  is  Air 
Force  Instruction  (AFI)  16-107,  Military  Personnel  Exchange  Program 
(MPEP).  Not  all  service-level  programs  are  governed  under  specific 
directives:  For  instance,  many  regional  seminars  have  no  directives  or 
instructions  whatsoever;  they  simply  use  Operations  and  Maintenance 
resources  under  the  authorities  that  govern  the  use  of  those  funds. 
Other  security  cooperation  programs  and  activities  derive  their  author¬ 
ity  from  memorandums  from  senior  leaders  or  simply  from  precedent. 

Given  the  wide  variety  of  authorities  that  govern  security  cooper¬ 
ation  programs,  it  is  nearly  impossible  to  find  directives  or  instructions 
that  provide  specific  insights  into  how  stakeholders  should  assess  their 
respective  programs  for  effectiveness  or  efficiency.  At  present,  the  GEF 
provides  the  only  real  assessment  guidance  available  to  the  COCOMs, 
services,  and  defense  support  agencies. 
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Indicators  and  the  Levels  of  Assessment 

In  order  to  be  able  to  measure  the  effects  of  programs  and  activities,  it 
is  necessary  to  first  establish  specific  assessment  indicators  that  are  con¬ 
sistent  over  time.  We  consider  three  main  types  of  indicators: 

•  Inputs  are  the  resources — money,  manpower,  etc. — that  are 
required  to  execute  an  event. 

•  Outputs  are  the  direct  products  of  an  event,  activity,  or  program. 

•  Outcomes  can  be  thought  of  as  the  effect  of  one  or  more  out¬ 
puts  on  the  target  audience,  or  changes  in  program  participants’ 
behavior,  knowledge,  skills,  status,  and/or  level  of  functioning. 

Metrics  are  observable,  measurable  evidence  of  inputs,  outputs,  and  out¬ 
comes.  The  actual  assessment  is  the  comparison  of  the  results  of  outputs 
and  outcomes  to  goals  and  to  more  specific,  measurable  objectives. 

For  program  assessment  purposes,  it  is  very  important  to  develop 
indicators  that  are  measurable  over  time,  to  show  change,  for  instance, 
in  the  capability  of  a  given  partner  country.  In  other  words,  the  indi¬ 
cators  themselves  should  not  change  from  year  to  year.  They  should 
change  only  if  the  desired  end-state  changes,  which  should  be  rare 
since  most  security  cooperation  goals — “improving  interoperability,” 
“building  partner  capability  and  capacity,”  and  “improving  the  U.S. 
operational  access  around  the  world” — are  enduring.  These  goals  were 
included  in  OUSD/P’s  Security  Cooperation  Guidance  and  are  now 
part  of  the  GEF  as  focus  areas. 

A  framework  is  necessary  to  sustain  the  explicit  focus  on  assess¬ 
ment  for  decisionmaking  and  to  connect  stakeholders  and  their  deci¬ 
sion  needs  with  specific  types  of  assessment.  The  study  team  found  the 
“hierarchy  of  evaluation”3  presented  in  Figure  3.2  to  be  useful.  For 
purposes  of  the  security  cooperation  program  assessment  process,  this 
hierarchy  is  best  thought  of  as  five  levels  of  assessment. 


3  This  term  comes  from  Richard  A.  Berk  and  Peter  H.  Rossi,  Thinking  About  Program 
Evaluation ,  Newbury  Park,  Calif.:  Sage  Publications,  1990. 
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Figure  3.2 
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The  hierarchy  divides  potential  evaluations  and  assessments 
into  five  nested  levels:  As  a  general  rule,  each  level  depends  on  assess¬ 
ments  at  the  lower  levels.  For  example,  positive  assessments  of  cost- 
effectiveness  (the  highest  level)  are  possible  only  if  supported  by  posi¬ 
tive  assessments  at  all  other  levels.  In  fact,  as  we  describe  later,  this 
structure  also  works  as  a  cycle.  However,  one  can  conceive  of  situa¬ 
tions  in  which  the  cost-effectiveness  calculation  is  favorable  even  where 
lower-level  assessment  are  not,  if  the  need  for  the  program  is  so  great 
that  even  modest  accomplishments  achieved  at  high  costs  would  be 
determined  to  be  cost-effective.  The  senior  organization,  OSD  in  this 
case,  should  logically  be  responsible  for  both  Level  1  (Is  there  a  need?) 
and  Level  5  (Does  this  program  do  better  than  the  others  from  a  com¬ 
parative  cost-effectiveness  perspective)? 

We  now  describe  each  of  the  levels  in  detail. 


Level  1.  Assessment  of  Need  for  Program 

Level  1  is  the  assessment  of  the  need  for  the  program  or  activity.  This  is 
where  evaluation  connects  most  explicitly  with  target  goals.  Evaluation 
at  this  level  focuses  on  the  problem  to  be  solved  or  goal  to  be  met,  the 
population  to  be  served,  and  the  kinds  of  services  that  might  contrib¬ 
ute  to  a  solution.  Assessment  questions  would  include  the  following: 
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•  What  are  the  nature  and  magnitudes  of  the  problems  to  be 
addressed? 

•  What  audience,  population,  or  targets  does  the  need  apply  to? 

•  What  existing  programs  or  activities  contribute  to  meeting  this 
goal  or  mitigating  this  problem? 

•  What  are  the  goals  and  objectives  to  be  met  through  policy  or 
program? 

Evaluation  at  the  needs  level  is  often  skipped  because  the  need  is 
assumed  to  be  obvious.  Where  such  a  need  is  genuinely  obvious  or  the 
policy  assumptions  are  clearly  right,  this  is  not  problematic.  Where 
need  is  not  obvious  or  goals  are  not  well  articulated,  however,  problems 
at  Level  1  can  complicate  assessment  at  each  higher  level. 

Level  2.  Assessment  of  Design  and  Theory 

The  assessment  of  concept,  design,  and  theory  is  the  second  level  in 
Figure  3.2.  Once  a  needs  assessment  establishes  that  there  is  problem 
or  policy  goal  worth  pursuing,  different  solutions  can  be  considered. 
This  is  where  theory  connects  types  of  security  cooperation  activities  to 
strategic  goals.  Assessment  at  this  level  focuses  on  the  design  of  a  policy 
or  program.  Analyses  of  alternatives  are  generally  evaluations  at  this 
level.  Questions  might  include  the  following: 

•  What  types  of  program  are  appropriate  to  meet  the  need? 

•  What  specific  services  should  be  provided,  in  what  quantity,  and 
for  how  long? 

•  How  can  these  services  best  be  delivered? 

•  What  outputs  need  to  be  produced? 

•  How  should  the  program  or  policy  be  organized  and  managed? 

•  What  resources  will  be  required  for  the  program  or  policy? 

•  Is  the  theory  specifying  certain  services  as  solutions  to  the  target 
problem  sound? 

Most  of  the  evaluation  questions  at  this  level  are  answered  based  either 
on  theory  or  on  previous  experience  with  similar  programs  or  activities. 
This  level  is  crucial:  If  program  design  is  based  on  poor  theory,  perfect 
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execution  may  still  not  result  in  the  desired  results.  Similarly,  if  theory 
does  not  actually  connect  the  types  of  activities  with  strategic  goals,  the 
program  may  accomplish  objectives  other  than  those  intended. 

Once  a  program  is  under  way,  design  and  theory  can  be  assessed 
firsthand.  Assessment  questions  at  this  level  for  an  ongoing  program 
could  include  the  following: 

•  Are  the  services  that  are  being  provided  adequate  in  duration  and 
quantity? 

•  Is  the  frequency  with  which  services  are  provided  adequate? 

•  Are  resources  sufficient  for  the  desired  execution? 

Note  that  assessment  at  this  level  is  not  about  execution  (“Are  the  ser¬ 
vices  being  provided  as  designed?”).  Execution  takes  place  at  Level  3. 
Design  and  theory  assessments  (Level  2)  seek  to  confirm  that  plans  are 
adequate  to  achieve  the  desired  objectives. 

Level  3.  Assessment  of  Process  and  Implementation 

Level  3  focuses  on  program  operations — in  particular,  on  the  execu¬ 
tion  of  the  elements  prescribed  by  the  theory  and  design  at  Level  2.  If 
the  design  is  inadequate,  a  program  can  be  perfectly  executed  but  still 
not  achieve  its  goals.  Conversely,  poor  execution  can  foil  the  most  bril¬ 
liant  design. 

Assessment  at  this  level  must  be  periodic  and  ongoing.  Just 
because  a  program’s  process  goals  are  being  met  at  one  point  in  time 
does  not  necessarily  mean  they  will  always  be  met  in  the  future.  In 
addition  to  measuring  process,  Level  3  evaluations  include  “outputs,” 
the  countable  deliverables  of  a  program.  This  level  is  the  place  for  a  lim¬ 
ited  assessment  of  efficiency:  Is  implementation  getting  the  most  pos¬ 
sible  out  of  the  available  resources?  Other  possible  questions  at  Level  3 
include  the  following: 

•  Were  necessary  resources  made  available? 

•  Are  the  intended  services  being  delivered  as  designed? 

•  Are  process  and  administrative  objectives  being  met? 

•  Is  the  program  being  managed  well? 
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•  Are  service  recipients  satisfied  with  their  service? 

•  Were  regulations  followed? 

•  Are  program  resources  being  used  and  consumed  as  intended? 

Outputs — the  direct  products  of  an  event,  activity,  or  program — 
can  be  identified  and  even  measured,  in  many  cases,  just  after  the  con¬ 
clusion  of  the  event.  To  illustrate,  if  the  activity  is  an  exercise,  perhaps 
executed  by  DTRA,  to  support  the  end-state  that  “allies  or  partners 
are  capable  partners  in  combating  weapons  of  mass  destruction,”  the 
output  indicators  shown  in  Figure  3.3  could  apply. 

The  RAND  Corporation  uses  a  basic  logic  model  to  develop  output 
indicators  for  types  of  security  cooperation  activities.4  This  model  helps 
program  managers  think  through  the  immediate  output  of  an  activity. 
Some  examples  of  common  output  indicators  for  BPC  programs  are 
number  of  personnel  trained,  equipment  transferred,  plans  prepared, 
and  number  of  meetings  held. 

As  stated  in  Chapter  Two,  the  workshop  participants  broke 
into  groups  to  discuss  building  output  measures  for  three  types  of 
activities: 


Figure  3.3 

Sample  Output  Indicators  for 
Exercises 
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•  Operational,  procedural, 
and  legal  problems 
identified 
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departmental/agency 
representation  included 
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For  more  information  on  this  logic  model,  see  Marquis  et  al.,  2006. 
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contacts,  train  and  equip,  and  classroom  education.  Participants  in  the 
contacts  group  developed  the  following  output  indicators:  number  of 
events,  type  of  event,  sophistication  of  the  event,  number  of  people  who 
attended,  appropriate  rank  representation,  and  agreements  reached.  In 
addition,  they  also  developed  process-related  indicators:  cost  of  the 
program  or  activity,  manpower  or  personnel  involved,  the  time  it  took 
to  perform  the  activity,  and  equipment  provided.  Participants  in  the 
contact  group  session  used  versions  of  these  indicators  at  the  strategic, 
operational,  and  tactical  levels  so  that  data  could  be  aggregated  from  the 
event  level  up  to  the  strategic  level  for  each  program.  The  participants 
in  the  train  and  equip  session,  in  contrast,  focused  on  output  mea¬ 
sures,  such  as  number  of  personnel  trained  in  accordance  with  United 
Nations  or  U.S.  standards,  necessary  equipment  available,  and  neces¬ 
sary  certification  obtained.  The  classroom  activities  session  focused  its 
output  indicators  on  the  education  process:  making  sure  that  classroom 
participants  are  learning  against  the  stated  objectives. 

Level  4.  Assessment  of  Outputs  and  Outcomes 

Level  4  is  near  the  top  and  concerns  outcomes  and  impact.  This  is 
where  outputs  are  translated  into  outcomes,  a  level  of  performance  or 
achievement.  Put  another  way,  outputs  are  the  products  of  program 
activities,  whereas  outcomes  are  the  changes  resulting  from  the  projects. 
This  is  the  first  level  of  assessment  at  which  solutions  to  the  problem 
that  originally  motivated  the  program  can  be  seen.  Questions  at  Level 
4  could  include  the  following: 

•  Do  the  services  provided  have  beneficial  effects  on  the  recipients? 

•  Do  the  services  provided  have  the  intended  effects  on  the 
recipients? 

•  Are  program  objectives  and  goals  being  achieved? 

•  Is  the  problem  at  which  the  program  or  activity  is  targeted 
improving? 

It  is  also  important  to  link  output  and  outcome  indicators  whenever 
possible.  To  take  the  example  of  an  exercise  to  improve  capabilities 
for  combating  weapons  of  mass  destruction,  as  discussed  in  Level  3, 
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the  following  output  and  outcome  indicators  could  apply,  as  shown  in 
Figure  3.4. 

It  is  important  to  select  those  indicators  that  best  apply  to  the  pro¬ 
grams  being  examined.  For  example,  if  a  program  focuses  on  exercises 
as  a  way  of  improving  the  border  security  capacity  of  a  specific  country, 
the  appropriate  output  indicators  may  include  whether  any  legal,  pro¬ 
cedural,  or  operational  problems  were  identified  during  the  course  of 
the  exercise,  whether  existing  capabilities  for  coalition  operations  were 
exercised,  and  whether  the  relevant  agencies  were  represented  at  the 
exercise.  Program  assessors  should  ask  questions  such  as  “Did  the  part¬ 
ner  country  send  a  representative  from  the  appropriate  ministry,  given 
the  topic  of  the  exercise?”  “Did  it  send  a  policy  representative  to  an 
exercise  focused  on  technical  or  scientific  aspects  of  combating  WMD 
threats?”  If  it  did  not,  U.S.  officials  might  question  whether  that  coun¬ 
try  is  interested  in  receiving  assistance  from  the  United  States  in  that 
format,  or  perhaps  whether  the  United  States  failed  to  communicate 
to  the  partner  the  types  of  participants  required  for  the  event.  In  any 
case,  something  would  definitely  appear  to  be  wrong,  perhaps  with  the 
design  of  the  event,  the  topic  relative  to  the  partner’s  interests,  the  U.S. 
government’s  ability  to  communicate  which  partner  agency  should 
send  representatives,  or  with  the  partner’s  willingness  to  engage. 

Figure  3.4 

Sample  Output  and  Outcome  Indicators  for  Exercises 
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As  shown  in  Figure  3.4,  appropriate  outcome  indicators  would 
include  whether  those  legal,  operational,  or  procedural  problems  iden¬ 
tified  as  an  output  of  the  event  were  later  resolved;  whether  the  capa¬ 
bilities  exercised  during  the  event  were  successfully  deployed  in  a  coali¬ 
tion  operation,  if  that  was  the  intent;  and  whether  common  standards 
or  concepts  of  operation  were  adopted  or  mutual  understanding  on  the 
topic  of  the  exercise  was  advanced. 

Note  that  outcomes  are  meant  to  be  enduring  and  thus  are  often 
very  difficult  to  measure.  Outcome  indicators,  by  design,  focus  on  the 
longer  term.  Therefore,  the  linkages  between  output  and  outcome  indi¬ 
cators  must  be  tracked  over  time.  The  challenge  is  that  program  manag¬ 
ers  executing  an  activity  in  support  of  a  geographic  COCOM  require¬ 
ment  are  not  necessarily  responsible  for  tracking  those  outcome  results 
over  time.  This  responsibility  often  lies  primarily  with  the  COCOM 
and  the  component  commander,  particularly  if  the  COCOMs  are 
responsible  for  developing  the  policy  outcomes  for  the  event.  Never¬ 
theless,  program  managers  should  be  aware  of  the  changes  that  may 
have  resulted  from  the  application  of  their  programs.  These  are  the 
data  inputs  that  help  program  managers  and  other  decisionmakers  to 
determine  if  the  program  and  the  specific  activities  should  be  contin¬ 
ued  as  is,  altered  in  some  way  (e.g.,  such  as  changing  the  scenario  in  an 
exercise  or  targeting  a  different  capability  in  a  partner  country),  or  cut 
due  to  insufficient  performance. 

For  outcomes  too,  RAND  used  a  basic  logic  model.  Examples  of 
common  outcome  indicators  for  BPC  programs  include  the  number  of 
personnel  who  were  trained  and  then  established  training  programs  for 
third  countries  or  additional  units  in  their  own  military,  whether  the 
equipment  transferred  enabled  the  partner  military  to  assist  U.S.  oper¬ 
ations,  whether  plans  prepared  were  translated  into  the  budgeting  and 
acquisition  process,  and  how  many  meetings  held  with  partner  coun¬ 
tries  resulted  in  a  memorandum  of  understanding  that  has  improved 
U.S.  access  to  a  new  region. 

Workshop  participants  in  the  contacts  group  developed  the  fol¬ 
lowing  outcome  indicators: 


46  A  Framework  to  Assess  Programs  for  Building  Partnerships 


•  Events  became  increasingly  sophisticated. 

•  Participants  gained  knowledge  to  conduct  their  own  events. 

•  Event  participants  were  enabled  to  build  new  organizational 
capacity. 

•  Agreements  were  implemented. 

•  The  contacts  produced  the  desired  effect  (e.g.,  improved  border 
security). 

In  addition,  the  contacts  group  designed  measures  of  effectiveness 
to  answer  the  following  questions:  Does  the  activity  or  event  meet  the 
country  objectives?  Do  the  selected  events  and  projects  support  theater 
strategy  and  U.S.  goals  in  the  region?  Finally,  does  the  collection  of 
events  at  the  regional  level  support  U.S.  goals  at  the  DoD  and  national 
level? 

The  participants  in  the  train  and  equip  session,  by  contrast, 
focused  on  outcome  measures  such  as  the  following: 

•  The  country  sustained  a  capability  to  conduct  desired  operations. 

•  The  partner  country’s  equipment  and  training  needs  have  been 
met. 

•  Trained  personnel  remained  in  service  a  number  of  years  after 
receiving  training. 

•  The  country  is  able  to  export  capability  to  others. 

The  classroom  activities  session,  on  the  other  hand,  focused  its  outcome 
indicators  on  how  the  education  gained  by  classroom  participants  helps 
the  United  States  reach  desired  goals.  For  example, 

•  Alumni  transferred  knowledge  gained  to  others. 

•  Alumni  worked  to  change  or  write  new  laws  and  policies,  build 
new  organizations,  or  capabilities. 

•  Alumni  became  willing  to  host  a  forum  on  a  new  topic  and  to 
provide  feedback  on  U.S.  policy  in  the  region. 
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Level  5.  Assessment  of  Relative  Cost-Effectiveness 

The  assessment  of  relative  cost-effectiveness  sits  at  the  top  of  the  levels 
of  assessment,  at  Level  5.  This  is  “bang  for  the  buck.”  Only  when 
desired  outcomes  are  at  least  partially  observed  can  efforts  be  made  to 
assess  their  cost-effectiveness;  that  is,  before  you  can  measure  “bang  for 
the  buck,”  you  have  to  be  able  to  measure  “bang.”  Evaluations  at  this 
level  are  often  the  most  attractive  in  terms  of  the  bottom  line,  but  they 
depend  heavily  on  lower  levels  of  evaluation  and  can  be  complicated 
to  measure  when  resource  flows  are  unclear  or  outside  factors,  perhaps 
unforeseen,  affect  outcomes.  Assessments  at  this  level  can  provide  feed¬ 
back  inputs  for  policy  decisions  that  are  primarily  based  on  the  lower 
levels.  For  example,  if  cost-efficiency  target  levels  are  not  being  met, 
cost  data  (Level  5)  in  conjunction  with  process  data  (Level  3)  can  be 
used  to  streamline  the  process  or  otherwise  selectively  reduce  costs. 
Possible  Level  5  research  questions  include: 

•  How  efficient  is  resource  expenditure  relative  to  outcome? 

•  Is  the  cost  reasonable  relative  to  the  magnitude  of  benefits? 

•  Could  alternative  approaches  yield  comparable  benefit  at  lower 

cost? 

Assessment  Functions 

At  each  of  the  levels  of  assessment,  there  are  four  functional  assessment 
roles  that  DoD  and  other  agencies  can  and  do  perform  with  respect 
to  security  cooperation  programs,  whether  they  are  managed  by  the 
services;  outside  the  services,  for  instance  by  the  COCOMs  or  OSD; 
or  by  the  State  Department.  In  some  instances,  these  functions  are 
clearly  spelled  out  in  laws,  policy  directives,  and  program  instructions. 
In  other  cases,  they  must  be  inferred  by  taking  into  account  the  char¬ 
acter  of  the  organization  and  the  extent  of  its  de  jure  and  de  facto  deci¬ 
sionmaking  authority.  The  following  are  definitions  of  the  four  assess¬ 
ment  functions. 

Data  Collector.  According  to  standards  set  by  the  assessor  organi¬ 
zation,  the  data  collector  organization  is  responsible  for  collecting  and 
aggregating  data  from  internal  and  external  sources  for  a  particular 
level  of  programmatic  assessment. 
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Assessor.  This  organization  is  responsible  for  setting  data  collec¬ 
tion  standards  for  a  particular  kind  of  programmatic  assessment  and 
for  evaluating  programs  using  methods  suitable  for  the  types  of  assess¬ 
ment  it  performs. 

Reviewer.  This  organization  is  responsible  for  helping  assessors 
develop  data  collection  standards  and  evaluation  methods  appropri¬ 
ate  for  the  level  of  assessment  for  which  they  are  responsible.  It  is  also 
responsible  for  conducting  periodic  inspections  or  audits  to  ensure  that 
program  assessments  are  being  properly  executed. 

Integrator.  This  organization  is  responsible  for  organizing  and 
synthesizing  programmatic  assessments  to  meet  DoD  requirements  for 
the  GEF,  the  Capabilities  Portfolio  Management  System,  and  the  plan¬ 
ning,  programming,  budgeting,  and  execution  (PPBE)  process. 

It  should  be  noted  that  these  assessment  roles  are  intended  to 
guide  assessment  behavior,  not  overly  constrain  it,  because  DoD  orga¬ 
nizations  may  play  a  variety  of  security  cooperation  assessment  roles 
depending  on  the  category  of  program  and  the  level  of  assessment 
under  discussion.  Among  the  many  assessment  stakeholders,  some 
are  responsible  for  playing  a  role  in  the  program  assessment  process 
directly;  others — such  as  the  Office  of  Management  and  Budget  and 
Congress — are  consumers  of  the  assessments  themselves.  If  the  pro¬ 
gram  has  an  accompanying  directive  that  specifies  the  responsibilities 
of  the  various  stakeholders,  assigning  assessment  roles  is  much  easier. 


Summary 

This  chapter  has  described  the  five  key  elements  of  a  program-level 
assessment  framework  and  illustrated  them  with  examples.  Most 
of  the  elements — strategic  guidance,  programs,  stakeholders,  and 
authorities — are  fairly  straightforward.  However,  the  fifth  element, 
levels  of  assessment  and  specific  roles,  is  less  so.  Breaking  down  the  com¬ 
ponents  of  the  program  and  discussing  stakeholder  roles  and  responsi¬ 
bilities  as  a  team  can  help  stakeholders  identify  who  should  be  assessing 
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which  aspects  of  the  program  and  why,  according  to  decisions  they  are 
required  to  make.  The  following  chapter  takes  the  next  step:  develop¬ 
ing,  integrating,  and  illustrating  the  various  elements  of  the  program 
assessment  framework. 


CHAPTER  FOUR 


Implementing  the  Assessment  Framework 


This  chapter  presents  an  approach  for  implementing  the  five  elements 
of  the  OUSD/P  security  cooperation  assessment  framework  that  was 
introduced  in  Chapter  Three.  The  approach  is  specifically  focused  on 
the  OUSD/P-managed  programs  and  is  tailored  to  OUSD/P  program 
assessment  requirements.  It  takes  into  account  the  various  security 
cooperation  programs  in  which  OUSD/P  is  involved,  the  different 
kinds  of  assessment  that  are  possible  and  consistent  within  the  intent 
of  the  GEF,  the  supporting-supported  relationships  that  might  be  built 
to  make  such  assessments  possible,  and  the  data  that  would  have  to  be 
collected  about  specific  security  cooperation  programs  in  order  to  con¬ 
duct  useful  assessments. 

The  chapter  begins  by  addressing  the  critical  issue  of  stakehold¬ 
ers’  roles  in  the  assessment  process.  It  then  describes  basic  principles 
for  matching  organizations  to  assessment  roles  and  how  these  prin¬ 
ciples  might  generally  apply  to  OUSD/P-managed  security  coopera¬ 
tion  programs.  In  the  absence  of  a  program  directive  that  provides  spe¬ 
cific  responsibilities  to  the  various  stakeholders,  broad  selection  criteria 
can  be  useful  in  helping  to  think  through  the  appropriate  assessment 
roles  of  each  of  the  stakeholders.  The  chapter  then  examines  ways  in 
which  these  general  rules  might  have  to  be  modified  to  conform  to  the 
organizational  circumstances  of  specific  OUSD/P  programs.  Finally,  it 
describes  a  standardized  but  flexible  format  for  conducting  assessments 
at  each  level  of  the  assessment  hierarchy,  including  a  list  of  basic  ques¬ 
tions  and  associated  data  requirements. 
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Stakeholder  Assessment  Roles 

As  noted  earlier,  OSD  and  other  DoD  organizations  have  been  con¬ 
ducting  assessments  of  security  cooperation  activities  for  some  time — 
evaluating  exercises,  surveying  participants  for  their  opinions  about  the 
value  and  utility  of  the  activities,  and  trying  to  ascertain  the  degree  to 
which  those  activities  are  adhering  to  relevant  guidance  and  directives. 
Thus,  the  business  of  assessment  is  not  new  to  DoD.  Instead,  DoD’s 
goal  should  be  to  integrate  and,  where  necessary,  expand  existing  capa¬ 
bilities  to  enable  a  more  comprehensive  approach  to  assessing  BPC  and 
other  security  cooperation  programs. 

As  this  monograph  has  stressed,  all  programs  have  stakehold¬ 
ers,  and  each  stakeholder  enjoys  specific  responsibilities  and  different 
points  of  view  with  regard  to  a  program.  Furthermore,  the  levels  of 
assessment,  as  outlined  in  Chapter  Three,  help  determine  stakeholder 
assessment  roles,  just  as  the  authorities  of  stakeholders  influence  their 
responsibilities.  These  responsibilities,  in  turn,  shape  the  decisions  that 
fall  within  the  purview  of  the  stakeholder.  That  process  alone  does  not 
determine  the  substantive  views  of  any  stakeholder,  but  it  does  shape 
the  kinds  of  decisions  they  make. 

In  general,  given  the  relevant  authorities  as  outlined  in  Chapter 
Three,  the  highest  levels  of  leadership  within  DoD,  typically  within 
OSD,  will  be  responsible  for  the  critical  decisions  about  needs  for 
programs  and  for  their  design — that  is,  how  a  program  is  supposed 
to  lead  to  various  goals  and  end-states — as  well  as  for  assessing  cost- 
effectiveness  across  programs.  For  their  roles,  service  and  COCOM 
stakeholders  should  concentrate  on  matters  of  process  and  implemen¬ 
tation,  as  well  as  on  outcomes  and  impacts.  Again,  however,  none  of 
these  roles  is  cast  in  stone:  Consistency  across  programs  needs  to  be 
balanced  with  flexibility,  given  the  differences  in  the  programs. 
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Implementing  Security  Cooperation  Assessments 

Matching  Organizations  to  Assessment  Roles 

Legal  authorities,  set  forth  in  Title  10  and  Title  22  of  the  U.S.  Code, 
establish  the  principal  departmental  divisions  of  labor,  but  Title  10, 
especially,  gives  DoD  considerable  leeway  on  how  to  manage  the  pro¬ 
grams  within  its  domain.  As  stressed  earlier,  strategy  and  planning 
documents,  such  as  the  GEF,  describe  the  ends,  ways,  and  means  of 
security  cooperation  for  DoD.  However,  they  say  little  about  program 
execution,  including  assessment.1  Many  security  cooperation  programs 
have  accompanying  directives  or  operating  instructions  that  specify  the 
program’s  objectives,  how  resources  are  allotted  and  expended,  and  the 
various  stakeholder  responsibilities.  A  review  of  those  directives  and 
instructions,  depending  on  how  detailed  they  are,  can,  in  most  cases, 
make  assigning  assessment  roles  fairly  straightforward.  However,  not 
all  programs  have  associated  directives  or  operating  instructions.  Many 
programs,  such  as  the  Warsaw  Initiative  Fund,  are  governed  only  by 
broad  Title  10  guidance,  specifically,  10  U.S.C.  1051  and  10  U.S.C. 
168. 2  In  the  absence  of  more  specific  directives  or  instructions,  broad 
selection  criteria  can  be  helpful  in  thinking  through  the  appropriate 
assessment  roles  of  each  of  the  stakeholders. 

Because  a  range  of  DoD  organizations  already  perform  assess¬ 
ments  of  certain  aspects  of  security  cooperation,  OUSD/P  should 
be  able  to  determine  the  right  organizations  to  assist  it  in  evaluating 
the  security  cooperation  programs  it  manages  in  accordance  with  the 
assessment  framework  developed  here.  In  many  cases,  the  issue  will  be 
integrating  and  aligning  the  assessments  that  are  already  being  done. 
In  other  cases,  the  issue  may  be  assigning  new  responsibilities  to  orga¬ 
nizations  that  are  best  suited  to  perform  a  particular  kind  of  assess¬ 
ment.  In  a  few  cases,  there  may  be  a  need  to  establish  a  new  assessment 


1  For  example,  the  GEF  states  that  the  services  must  provide  output-oriented  assessments  of 
the  programs  they  conduct  in  support  of  the  COCOMs.  But  the  GEF  does  not  provide  the 
details  on  how  these  programs  should  be  assessed  over  time. 

2  Department  of  Defense,  Office  of  the  Inspector  General,  Joint  Warfighting  and  Readiness: 
DoD  Execution  of  the  Warsaw  Initiative  Program  (D-2005-085),  Washington,  D.C.,  July  1, 
2005. 
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function  or  organization  when  no  existing  organization  seems  right  for 
a  certain  role. 

As  has  been  emphasized,  existing  laws  and  directives  rarely  spell 
out  assessment  roles  in  detail,  and,  in  many  cases,  more  than  one  orga¬ 
nization  can  make  a  plausible  claim  for  a  particular  assessment  assign¬ 
ment.  Security  cooperation  program  instructions,  when  they  exist,  do 
establish  specific  managerial  responsibilities,  but  they  are  generally 
silent  on  how  programs  should  be  evaluated. 

That  said,  established  authorities  do  give  certain  DoD  organiza¬ 
tions  implied  assessment  responsibilities.  For  example,  Section  1206 
requires  the  State  Department  and  DoD  to  jointly  formulate  programs 
and  coordinate  implementation.  This  requirement  has  led  to  a  process 
wherein  1206  Program  proposals  are  submitted  by  the  COCOMs  and 
the  Chiefs  of  Mission,  are  principally  evaluated  by  OUSD/P  Partner¬ 
ship  Strategy  and  the  State  Department’s  Political-Military  Bureau, 
and  are  ultimately  approved  by  the  Secretary  of  Defense  and  the  Secre¬ 
tary  of  State  unless  Congress  raises  an  objection.  This  process  implies 
important  roles  for  the  COCOMs  and  OSD  in  determining  program 
need — the  first  as  a  collector  of  data  pertinent  to  the  needs  assessment 
and  the  second  as  the  actual  assessor  of  those  data.  To  use  another 
example,  the  reporting  requirements  to  Congress  for  the  Combat¬ 
ing  Terrorism  Fellowship  Program,  under  10  U.S.C.  2249c,  imply  an 
important  role  for  the  CTFP  program  management  office  to  collect  the 
data  necessary  to  assess  the  program’s  design,  execution,  outcome,  and 
cost-effectiveness. 

Despite  that  guidance,  in  many  instances  assigning  specific  assess¬ 
ment  responsibilities  to  particular  DoD  organizations  will  require 
looking  beyond  relevant  laws,  policies,  and  regulations.  In  particular, 
it  is  important  for  OSD  officials  to  pay  close  attention  to  an  organizations 
capabilities — in  terms  of  funding,  manpower,  expertise,  proximity,  and 
opportunity — as  well  as  an  organization’s  objectivity,  specifically  the 
extent  of  its  interest  in  specific  assessment  results.  For  example,  despite 
its  broad  security  cooperation  oversight  responsibilities,  OUSD/P  cur¬ 
rently  has  few  personnel  trained  to  conduct  assessments,  and  it  lacks 
the  resources  to  do  a  full  range  of  programmatic  assessments.  OUSD/P 
needs  to  develop  the  capacity,  as  described  below,  to  work  with  DSCA 
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in  integrating  the  assessments  of  other  DoD  organizations  to  satisfy 
the  requirements  of  the  GEF,  the  Comprehensive  Joint  Assessment,  the 
Capabilities  Portfolio  Management  System,  and  the  PPBE  process,  not 
to  mention  the  demands  of  the  U.S.  Congress. 

Organizational  Assignments  and  Stakeholder  Role  Selection  Criteria 

Many  DoD  organizations  might  serve  as  data  collectors,  assessors, 
reviewers,  and  integrators  for  OUSD/P-managed  security  coopera¬ 
tion  programs  on  the  five  levels  of  assessment  decisions.  A  key  goal  is 
to  inject  greater  objectivity  into  the  overall  assessment  process,  thus 
moving  away  from  the  current,  largely  self-assessment  approach  to 
BPC  and  security  cooperation  programs.  Especially  in  the  absence  of 
directives  and  instructions,  the  following  guidelines  should  inform  the 
process  of  assigning  various  stakeholders  with  assessment  roles: 

•  Delineate  assessment  responsibilities  across  several  stakeholders  to 
account  for  differing  levels  of  organizational  authority  and  exper¬ 
tise  and  to  inject  as  much  objectivity  into  the  process  as  possible. 

•  Identify  a  single  organization  with  a  close  connection  to  the  pro¬ 
gram  at  hand  that  should  be  ultimately  responsible  for  gathering 
and  collating  assessment  data,  although  data  collection  will  often 
involve  a  number  of  individuals  and  organizations  from  different 
parts  of  DoD  (and  even  outside  DoD). 

•  Recognize  that  in  some  cases,  the  data  collector  and  the  assessor 
will  be  the  same  individual;  more  likely,  these  positions  will  be 
held  by  persons  within  the  same  organization. 

•  Ensure  that  the  assessor  and  the  reviewer  are  not  the  same  person, 
although  they  may  be  within  the  same  organization  (however, 
this  is  not  ideal). 

•  Ensure  that  integrators  and  especially  reviewers  pay  careful  atten¬ 
tion  to  what  data  are  collected  and  what  attributes  are  selected  as 
outputs  and  outcomes,  lest  attributes  be  designed  to  fit  what  the 
program  has  done,  not  necessarily  the  program’s  goals. 

•  Maintain  strong  linkages  between  integrators  and  program  stake¬ 
holders  to  develop  as  much  standardization  as  possible  and  as 
much  clarity  on  best  practices  in  security  cooperation  assessment. 


56  A  Framework  to  Assess  Programs  for  Building  Partnerships 


In  addition,  integrators  should  develop  mechanisms  for  storing 
assessment  information  (so  that  it  is  available  to  as  wide  a  group  of 
program  stakeholders  as  possible)  and  synthesizing  this  informa¬ 
tion  for  various  decisionmaking  purposes. 

Data  Collector.  Once  a  program  has  been  established,  program 
managers  are  generally  in  the  best  position  to  gather  and  aggregate 
data  from  their  own  organizations,  the  combatant  commands,  country 
teams,  partner  countries,  and  others  regarding  program  need,  design, 
implementation,  outcome,  and  cost-effectiveness.  Therefore,  they  are 
best  suited  to  serve  as  the  primary  data  collectors.  Prior  to  the  estab¬ 
lishment  of  a  program,  information  relevant  to  program  need  would 
necessarily  have  to  be  collected  by  another  organization  with  respon¬ 
sibility  for  OUSD/P-managed  programs.  In  most  cases,  this  organiza¬ 
tion  would  be  OUSD/P  Partnership  Strategy. 

Assessor.  The  assessor  responsibility  should  probably  vary  depend¬ 
ing  on  the  type  of  assessment  that  is  being  performed.  For  instance, 
assessing  the  need  for,  outcome  of,  and  cost/benefit  of  OUSD/P  secu¬ 
rity  cooperation  programs  should  be  assigned  to  OUSD/P  Partnership 
Strategy  as  the  chief  proponent  for  these  programs  and  the  organiza¬ 
tion  that  sets  most  program  objectives.  Given  their  implementation 
role,  policy  overseers  and  particularly  program  managers  are  probably 
best  suited  to  assess  the  design  and  execution  of  most  OUSD/P  pro¬ 
grams,  using  input  from  the  COCOMs  and  country  teams. 

Reviewer.  The  role  of  program  reviewer  is  best  handled  by  inde¬ 
pendent  analytical,  audit,  or  inspection  agencies.  For  high-level  needs 
and  cost-benefit  analyses,  reviewers  should  probably  come  from  out¬ 
side  the  security  cooperation  community  to  ensure  the  integrity  of 
OSD’s  overall  program  evaluation  system.  OSD’s  Directorate  of  Pro¬ 
gram  Analysis  and  Evaluation  is  responsible  for  managing  the  program 
review  phase  of  the  entire  PPBE  system;  thus,  it  is  in  a  position  to  bal¬ 
ance  requirements  for  security  cooperation  and  non-security  coopera- 
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tion  programs.3  Other  possible  reviewers,  especially  in  the  area  of  cost- 
effectiveness,  may  include  the  OSD  comptroller. 

For  reviews  of  security  cooperation  program  design,  execution, 
and  outcome,  OUSD/P  might  consider  standing  up  a  new  audit  func¬ 
tion  staffed  with  experts  in  both  assessment  and  security  cooperation. 
Attached  to  OUSD/P  and  most  likely  funded  by  OUSD/P  BPC  and 
other  security  cooperation  programs,  this  function  could  be  organized 
topically — around,  say,  operational  capacity,  institutional  capacity, 
and  human  capacity.  It  could  be  composed  of  personnel  detailed  from 
OUSD/P,  DSCA,  and  other  government  organizations  via  Intergov¬ 
ernmental  Personnel  Agreements,  as  well  as  from  several  contractors. 

This  new  function  would  not  duplicate  the  data  collection,  assess¬ 
ment,  or  integration  functions  being  performed  by  program  managers, 
the  COCOMs,  DSCA,  or  other  organizations  within  OSD.  Rather,  it 
would  draw  from  existing  data  sources  and  audit  existing  assessment 
mechanisms.  Periodically,  OSD  could  sponsor  visits  by  mobile  assess¬ 
ment  teams — possibly  composed  of  U.S.  government  personnel,  host 
nation  representatives,  and  contractors — to  priority  countries  to  evalu¬ 
ate  ambiguous  or  conflicting  assessment  information  through  on-site 
inspections,  personal  interviews,  and/or  surveys  and  polling.4  To  be 
sure,  the  DoD  Inspector  General  (IG)  currently  performs  an  auditing 
function  for  OUSD/P  security  cooperation  programs  on  a  infrequent 
basis.  Such  IG  insights  are  extremely  important  to  capture  in  any  new, 
more  frequent,  auditing  function  that  OUSD/P  may  wish  to  establish 
specifically  for  its  BPC  programs.  Any  new  process  should  comple¬ 
ment,  not  duplicate,  the  IG  function,  while  recognizing  that  IG  inspec¬ 
tions  are  by  their  nature  episodic,  not  continuing,  and  historically  have 
focused  on  abuse  or  error,  not  on  maximizing  performance. 

Integrator.  The  role  of  integrating  the  various  program  assess¬ 
ments  should  probably  fall  to  OUSD/P  Partnership  Strategy  and 


3  Department  of  Defense,  “Director,  Program  Analysis  and  Evaluation  (PA&E),”  DoD 
Directive  5141.01,  Washington,  D.C.,  March  16,  2006. 

^  For  assistance  in  implementing  this  security  cooperation  assessment  framework,  OSD 
might  want  to  consider  seeking  the  services  of  the  Federal  Consulting  Group — a  franchise 
within  the  U.S.  Department  of  the  Treasury  with  a  proven  track  record  of  improving  the 
management  of  large,  high-profile  U.S.  government  programs. 
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DSCA.  As  the  security  cooperation  program  assessment  integrator, 
OUSD/P  Partnership  Strategy  would  be  well  positioned  to  fulfill  its 
role  as  manager  of  the  Building  Partnerships  Portfolio  within  the  new 
Capabilities  Portfolio  Management  System  and,  potentially,  to  perform 
a  comparable  function  within  the  PPBE  process. 

Table  4.1  illustrates  how,  using  the  above  logic,  assessment  roles 
may  be  assigned  to  program  stakeholders.  These  assignments  are 
generic  and  may  not  fit  the  needs  or  requirements  of  particular  pro¬ 
grams.  Although  some  of  these  ideas  emerged  during  the  assessment 
workshop,  they  should  be  further  reviewed  by  the  DoD  security  coop¬ 
eration  community  before  they  are  proposed  for  approval  by  OUSD/P 
policymakers.  These  proposed  organizational  assignments  are  only  a 
first  step  in  establishing  an  integrated  structure  of  assessment  roles  and 
responsibilities  that  could  eventually  encompass  service-  and  COCOM- 
managed  security  cooperation  programs,  as  well  as  security  assistance 
programs  overseen  by  the  State  Department. 


Table  4.1 

Illustrative  Assessment  Roles  for  OUSD/P-Managed  Programs 


Assessment 

Decision 

Data  Collector 

Assessor 

Reviewer 

Integrator 

Need  for 
program 

OUSD/P  PS/ 
program  manager 

OUSD/P  PS 

OSD  (PA&E) 

OSD  PS/DSCA 

(via  Building 

Partnerships 

(BP)  Capability 

Portfolio 

Manager 

(CPM) 

Design  and 
theory 

Program  manager 

Program 

manager 

New  OSD  SC 
auditor? 

OUSD/P  PS/ 
DSCA 

(via  BP  CPM) 

Process  and 
implementation 

Program  manager 

Program 

manager 

New  OSD  SC 
auditor? 

OUSD/P  PS/ 
DSCA 

(via  BP  CPM) 

Outputs  and 
outcomes 

Program  manager 

OUSD/P  PS 

New  OSD  SC 
auditor? 

OUSD/P  PS/ 
DSCA 

(via  BP  CPM) 

Cost- 

effectiveness 

Program  manager 

OUSD/P  PS 

OSD  PA&E 

OUSD/P  PS/ 
DSCA 

(via  BP  CPM) 
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Specific  Program  Assignments 

The  question  remains  as  to  whether  the  generic  assessment  roles  and 
responsibilities  shown  in  Table  4.1  are  appropriate  for  specific  OUSD/ 
P-managed  programs.  In  other  words,  can  OUSD/P  establish  a  consis¬ 
tent  and  comprehensive  assessment  framework  that  encompasses  the 
range  of  stakeholders,  authorities,  goals,  and  types  of  activities  asso¬ 
ciated  with  OUSD/P  security  cooperation  programs?  An  examina¬ 
tion  of  four  OUSD/P-managed  programs  and  funding  sources — the 
Warsaw  Initiative  Fund,  the  DoD  Regional  Centers  for  Security  Stud¬ 
ies,  the  National  Guard  State  Partnership  Program,  and  the  Section 
1206  BPC  authority — suggests  that  the  answer  is  a  qualified  yes.  The 
generic  framework  mostly  conforms  to  the  program  roles  and  responsi¬ 
bilities  currently  being  performed  by  DoD  agencies,  or  at  least  does  not 
contravene  them.  However,  there  are  significant  exceptions,  indicating 
again  that  any  framework  should  not  be  applied  in  a  cookie-cutter 
fashion  without  considering  the  unique  characteristics  and  require¬ 
ments  of  each  program. 

Warsaw  Initiative  Fund.  The  managers  of  various  WIF  programs 
are  responsible  for  planning  the  activities  actually  conducted  by  other 
DoD  organizations,  such  as  the  U.S.  Army  Corps  of  Engineers  and  the 
U.S.  Army  International  Affairs  Office,  as  well  as  by  private  contractors 
for  the  Civil  Military  Emergency  Preparedness  Program.  Thus,  it  seems 
appropriate  that  these  managers  be  required  to  gather  data  on  program 
design  and  execution.  Whether  they  should  also  collect  outcome  and 
cost-benefit  data  is  less  clear  since  they  have  less  control  or  insight  into 
these  results.  That  said,  a  central  data  collection  point  for  WIF  pro¬ 
grams  seems  appropriate  from  an  organizational  standpoint.  Program 
managers’  planning  and  resource  management  role  also  makes  them 
the  logical  assessors  of  WIF  program  design  and  execution. 

The  responsibilities  of  OUSD/P  and  DSCA  for  establishing  WIF 
goals  and  controlling  resources  make  them  the  best  candidates  for 
assessing  program  needs  and  cost-effectiveness  based  on  inputs  from 
the  program  managers,  implementing  agencies,  and  the  COCOMs. 
The  proposed  assignment  of  OSD  PA&E  as  the  reviewer  of  OUSD/P’s 
assessments  of  WIF  program  needs  and  cost-effectiveness  appears  to 
conform  to  that  organization’s  current  charter.  However,  standing  up 
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an  auditing  agency  within  OUSD/P  to  review  other  WIF  assessments 
would  most  likely  require  a  reallocation  of,  or  increase  in,  program 
funding.  Finally,  the  overall  managerial  responsibility  of  OUSD/P  and 
DSCA  for  the  Warsaw  Initiative  Fund  makes  these  two  agencies  the 
logical  integrators  of  WIF  program  assessments. 

Regional  Centers  for  Security  Studies.  The  Regional  Centers 
both  plan  and  execute  security  cooperation  activities.  Thus  the  func¬ 
tion  of  collecting  data  for  security  cooperation  assessments  can  largely 
be  done  internally  without  having  to  reach  out  to  numerous  imple¬ 
menting  agencies.  In  addition,  two  of  the  centers  (the  George  C.  Mar¬ 
shall  Center  and  the  Asia-Pacific  Center)  already  have  robust  systems 
for  assessing  the  design  and  execution  of  courses  and  other  activities. 
Unlike  with  WIF,  the  COCOMs  (not  OUSD/P  or  DSCA)  are  in  the 
best  position  to  evaluate  whether  the  Regional  Centers  are  meeting 
outcome-oriented  objectives  set  forth  in  the  COCOMs’  theater  cam¬ 
paign  plans.  OUSD/P  and  DSCA  play  a  similar  top-level  policy  and 
resource  management  role  with  respect  to  WIF  and  the  Regional  Cen¬ 
ters.  Thus,  a  similar  argument  can  be  made  for  assigning  them  the  role 
of  joint  assessment  integrators  in  both  cases. 

National  Guard  State  Partnership  Program.  In  the  case  of  the 
National  Guard  State  Partnership  Program,  the  responsibility  for  over¬ 
all  policy  and  resource  management  is  shared  between  OUSD/P  and 
the  National  Guard  Bureau’s  (NGB’s)  International  Affairs  Office. 
This  makes  the  NGB  (as  opposed  to  DSCA)  OUSD/P’s  logical  part¬ 
ner  in  integrating  SPP  assessments  from  lower-level  DoD  organiza¬ 
tions.  Likewise,  the  responsibility  for  assessing  program  outcomes 
probably  belongs  to  both  OUSD/P  and  the  NGB,  with  input  from  the 
COCOMs  and  the  country  teams.  The  SPP  data  collection  responsibil¬ 
ity  logically  belongs  to  the  States’  National  Guard  Adjutants  General, 
who  have  operational  control  over  the  National  Guard  units  partnering 
with  foreign  countries.  However,  the  fact  that  the  Adjutants  General 
report  to  state  governors,  rather  than  to  COCOM  commanders,  makes 
this  assignment  problematic.  Perhaps  this  data  collection  role  is  better 
given  to  a  National  Guard  liaison  office  within  the  COCOM  staff. 
This  office  could  also  coordinate  assessments  of  SPP  design  and  execu¬ 
tion  within  particular  theaters. 
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1206  Train  and  Equip  Program.  As  OUSD/P’s  partner  in  the  over¬ 
all  management  of  Section  1206  train-and-equip  funding,  the  Depart¬ 
ment  of  State’s  Political-Military  Bureau  should  probably  share  the 
responsibility  for  integrating  the  assessments  of  the  projects  financed 
under  this  joint  authority.  Because  of  their  goal-setting  responsibili¬ 
ties,  these  two  agencies  should  also  assess  1206  Program  outcomes. 
Individual  1206  project  managers  should  collect  the  data  required 
for  various  kinds  of  assessment.  In  addition,  the  program  managers 
should  evaluate  the  adequacy  of  project  design  and  process  in  coopera¬ 
tion  with  the  service  components  and  country  team  elements  respon¬ 
sible  for  project  execution.  The  Joint  Staff/J-5  may  have  an  assessment 
role  with  respect  to  1206  projects  because  this  organization  is  involved 
in  the  selection  of  projects  and  in  the  assessment  trips  to  the  various 
countries.  Ultimately,  however,  OSD  has  the  lead  as  program  manager 
of  the  1206  Program,  so  J-5  involvement  should  be  on  a  case-by-case 
basis,  as  appropriate,  and  in  cases  where  Joint  Staff  program  assessment 
expertise  is  made  available  to  OSD  for  this  purpose. 


Assessment  Questions  and  Data  Requirements 

Once  OUSD/P  has  clarified  the  responsibilities  of  the  security  coop¬ 
eration  stakeholders,  it  should  develop  a  common,  though  flexible, 
format  for  actually  conducting  program  assessments.  At  each  level  of 
the  process,  there  are  basic  assessment  questions,  the  answers  to  which 
will  vary  depending  upon  the  program’s  nature,  the  authorities  of  the 
stakeholders,  and  so  on.  Given  that  programs  are  the  unit  for  analysis, 
there  needs  to  be  a  mechanism  that  can  produce  program-level  answers 
to  these  questions  and  aggregate  individual  assessments  from  individ¬ 
ual  program  events  and  activities  over  time,  perhaps  several  years,  to 
produce  program-level,  time-series  insights  about  the  program’s  per¬ 
formance.  The  time-series  data  should  reveal  trends  that  will  allow 
OUSD/P  to  determine  whether  the  trajectory  of  individual  security 
cooperation  programs  and  the  trajectory  of  the  relationship  with  the 
partner  countries  are  consistent  (generally  positive,  stable,  or  generally 
negative). 
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Assessments  like  these  can  prove  complicated,  so  the  supported 
organizations  conducting  the  assessment  have  an  obligation  to  develop 
a  careful  assessment  design  and  to  stick  with  it,  while  the  supporting 
organizations  have  an  obligation  to  archive  the  essential  data  to  fuel  the 
assessment,  paying  attention  to  data  counting  rules:  individual  attend¬ 
ees  versus  whole  classes,  hours  of  events  versus  days  of  events,  compa¬ 
rable  activities,  etc.,  so  that  assessments  conducted  across  several  years 
will  employ  consistent  metrics. 

Table  4.2  lists  the  basic  assessment  questions  and  the  types  of  sup¬ 
porting  data  that  would  have  to  be  maintained  in  order  to  answer  those 


Table  4.2 

Basic  Assessment  Questions  and  Supporting  Data 


Questions 

Supporting  Data 

Need  for  the  Program 

Is  demand  for  the  program  growing, 
steady,  or  shrinking? 

Records  of  demand  over  time:  requests 
to  participate,  letters  of  agreement, 
letters  of  intent,  etc. 

Among  all  OUSD/P  programs,  where 
does  this  one  rank? 

Knowledge  of  overall  programs  and  the 
priority  attached  to  each 

If  OUSD/P  faces  budget  cuts,  is  this 
program  a  bill-payer  or  a  priority  for 
protection? 

Knowledge  of  overall  programs  and  the 
priority  attached  to  each 

Are  there  other  programs  that  produce 
the  same  benefits  with  the  same  partners? 

Knowledge  of  overall  programs,  their 
participants,  and  their  benefits 

If  so,  what  are  the  programs'  relative 
cost-effectiveness? 

Cost-benefit/cost-effectiveness  data  for 
all  programs 

Design  and  Theory 

Does  logic  or  theory  lead  us  to  expect 
that,  given  the  inputs  to  the  program, 
we  should  expect  the  outputs  claimed 
for  the  program? 


Security  cooperation  guidance,  program 
documentation  describing  goals,  and 
expected  contributions  that  program 
outputs  will  make 


Do  assumptions  linking  program 
performance  to  security  cooperation 
focus  areas  appear  logical? 


Program  documentation  describing 
goals  and  expected  contributions  that 
program  outputs  will  make 


Do  the  claimed  associations  between 
security  cooperation  focus  areas  and 
regional/functional  end-states  seem 
logically  consistent? 

Has  program  produced  desired  outputs 
or  outcomes  in  the  past? 


Program  documentation  describing 
goals  and  expected  contributions  that 
program  outputs  will  make;  knowledge 
of  relevant  end  states 

Past  performance  data  for  program 
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Table  4.2 — continued 


Questions 


Supporting  Data 


Process  and  implementation 

Is  the  program  resourced  sufficiently  to  Demand  data,  resource  data  (personnel, 
perform  its  functions  and  activities  relative  materiel,  funding) 
to  demand  for  them? 


Does  the  program  meet  deadlines,  fill 
quotas,  and  otherwise  satisfy  performance 
and  administrative  standards? 

Does  the  program  observe  restrictions  and 
prohibitions  with  respect  to  technology 
transfers,  spending  constraints,  and 
prohibitions  associated  with  program- 
element  money? 

Is  program  execution  conducted  to  foster 
positive  impressions  among  its 
participants? 


Records  of  administrative  and 
operational  performance,  attendees, 
participants,  numbers  of  graduates 

Export/transfer  authority 
documentation,  financial  records 


Exit  surveys  of  participants  collected 
over  time  to  support  time-series  analysis 


Outputs  and  Outcomes  of  the  Program 


Do  participants  leave  with  more  skill/  Entry  and  exit  testing  collected  over 

capacity  than  they  arrived  with?  time  to  support  time-series  analysis 


Is  partner  capability  in  the  program's  Time-series  data  on  partner  capabilities 

areas  growing,  stable,  or  declining?  Measures  of  performance  for  focus 

area: 

Operational  capacity:  partner  trend 
data  in  numbers  of  certified  units, 
operational  equipment 

Assurance  and  regional  confidence¬ 
building:  trends  in  confidence-  and 
security-building  measures,  security 
regimes,  security  cooperation 
activities  (frequency,  size,  number  of 
participants) 

Security  sector  reform:  trends  in 
adoption  of  U.S.  practices,  trends 
in  corruption  based  on  Security 
Assistance  Office  and  Defense 
Attache  Office  reports 
Exports  and  international 
collaboration:  trends  in  numbers 
of  orders  placed/filled,  size  and 
duration  of  deployments  with  U.S. 
forces 

National  and  multinational 
influence:  trends  in  numbers  of 
participating  countries,  numbers  of 
program  activities,  and  numbers  of 
participants  from  each  country 
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Table  4.2 — continued 


Questions 

Supporting  Data 

Cost-Effectiveness 

What  is  the  cost  per  unit  of  output? 

Cost  data,  data  on  units  of  output 

How  do  cost-effectiveness  data  compare 
with  other  security  cooperation  programs? 

Cost-effectiveness  data  on  other 
security  cooperation  programs 

What  is  return  on  investment  for  the 
program? 

Return  on  investment  data 

How  does  return  on  investment  compare 
with  other  programs? 

Cost  data  for  all  programs 

Do  any  other  programs  produce  the  same 
outputs  for  less  cost? 

Detailed  cost-process  information 

What  can  be  done  to  reduce  the  cost  per 
unit  of  output? 

Detailed  cost-process  data 

questions.  These  questions  are  suggestive  of  what  might  be  asked  of 
data  collectors  by  assessor  organizations  at  each  level  of  the  assessment 
hierarchy.  The  questions  could  be  modified  to  fit  the  specific  infor¬ 
mation  needs  of  the  assessing  organization.  The  key  principle  for  the 
assessing  organization  to  remember  is  that  it  must  ask  questions  whose 
answers  will  support  decisions  related  to  the  program  in  question. 

A  Proposed  OUSD/P  Assessment  Approach 

To  summarize  the  assessment  approach  described  thus  far  in  this  mono¬ 
graph,  we  propose  that  OUSD/P  consider  taking  four  basic  steps. 

First,  OUSD/P  should  coordinate  with  the  other  major  players  in  the 
DoD  security  cooperation  community  to  reach  a  consensus  regarding  the 
definitions  of,  and  linkages  among,  the  key  elements  of  the  program  assess¬ 
ment  framework.  This  consensus-building  effort  should  involve  work¬ 
ing  discussions  with  major  security  cooperation  stakeholders.  These 
discussions  could  be  incorporated  into  regularly  scheduled  security 
cooperation  meetings  sponsored  by  the  COCOMs  and  the  services. 
Participants  could  include  representatives  from  OUSD/P  Partner¬ 
ship  Strategy,  DSCA,  OUSD/P-managed  security  cooperation  pro¬ 
grams,  the  geographic  COCOMs,  and  service  headquarters  interna- 
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tional  affairs  offices.  Specifically,  these  working  groups  would  do  the 
following: 

•  Define  what  constitutes  a  program  for  assessment  purposes. 

•  Identify  key  stakeholders  for  each  OUSD/P  program. 

•  Associate  programs  with  security  cooperation  strategic  goals, 

focus  areas,  and  types  of  activities. 

Second,  OUSD/P  should  clarify  the  assessment  roles  and  responsi¬ 
bilities  for  each  program  stakeholder  by  level  of  assessment.  The  process 
of  assigning  supported  and  supporting  roles  to  specific  organizations 
should  be  undertaken  in  a  deliberate  manner,  program  by  program. 
Although  OUSD/P  Partnership  Strategy  should  have  final  say  over 
these  assignments,  its  decisions  should  be  made  only  after  considerable 
consultation  with  program  managers  and  other  stakeholders.  In  addi¬ 
tion,  although  it  should  take  into  account  general  principles  (such  as 
those  described  in  this  chapter)  when  matching  organizations  to  assess¬ 
ment  roles,  OUSD/P  should  modify  these  principles  as  necessary  to 
meet  the  requirements  of  particular  programs. 

Third,  OUSD/P  should  approve  appropriate  assessment  questions 
for  each  level  of  assessment  and  security  cooperation  method.  The  assessor 
organizations,  as  determined  in  the  preceding  step,  should  take  charge 
of  drafting  specific  program  assessment  questions — in  coordination 
with  the  lead  data  collector  when  these  two  organizations  are  differ¬ 
ent.  Once  approved,  these  questions  should  be  widely  disseminated, 
perhaps  as  an  annex  to  the  GEF  or  defense  agency  campaign  support 
plans. 

Finally,  OUSD/P  should  implement  a  comprehensive  security  coop¬ 
eration  assessment  framework  for  the  programs  that  it  directly  manages. 
Although  assessments  could  begin  all  at  once  across  the  range  of 
OUSD/P-managed  programs,  we  recommend  that  OUSD/P  select  one 
or  two  programs  with  resident  evaluation  expertise  as  test  cases  that 
could  be  used  to  help  resolve  any  unexpected  procedural  problems  that 
might  otherwise  confound  less  assessment-savvy  programs.  Figure  4.1 
illustrates  how  the  assessment  process  might  work  in  practice. 
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Figure  4.1 

A  Potential  Approach  to  Implementing  an  OUSD/P  Assessment  Framework 
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As  the  figure  suggests,  OUSD/P  could  select  its  security  coopera¬ 
tion  programs  from  among  the  categories  of  activities  and  assess  them 
at  all  levels  of  the  assessment  process.  OUSD/P  would  need  to  use  the 
measures  of  performance  at  the  level  of  outcome  and  impact  to  assess 
the  programs’  relative  contributions  toward  the  security  cooperation 
focus  areas.  OUSD/P  would  task  appropriate  subordinate  elements,  the 
data  collectors  identified  in  Table  4.1  above,  to  collect  and  provide  the 
necessary  data  to  the  assessing  organizations,  the  organizations  listed  in 
the  “assessor”  column  of  Table  4.1,  to  perform  the  actual  assessments. 
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Periodically,  the  specialized  reviewing  organizations  also  identified  in 
Table  4.1  would  check  the  methods  and  results  of  the  assessors. 

Once  the  assessments  were  completed  and  reviewed,  they  could 
be  passed  to  the  assessment  integrator,  probably  OUSD/P  Partnership 
Strategy  and  DSCA.  These  agencies  would  jointly  organize  and  pre¬ 
sent  the  assessments  and  recommendations  to  the  senior  decisionmak¬ 
ers  for  their  decisions  regarding  each  program  assessed.  The  entire  pro¬ 
cess  might  be  synchronized  to  support  the  annual  budget  cycle.  In 
following  this  assessment  procedure,  DoD  senior  leaders  would  have 
at  their  disposal  a  wealth  of  information  on  security  cooperation  pro¬ 
grams  that  would  allow  them  to  satisfy  the  requirement  for  assessment 
contained  in  the  GEF.  These  data  would  also  equip  the  leadership  to 
address  other  decisions  concerning  security  cooperation  in  times  where 
trade-offs  and  cutbacks  may  become  necessary  if  defense  budgets 
decline. 


Summary 

This  chapter  has  suggested  an  approach  to  implementing  the  OUSD/P 
security  cooperation  assessment  framework  proposed  in  this  mono¬ 
graph.  It  proposed  ways  to  overcome  the  impediments  to  sound  assess¬ 
ments  of  security  cooperation  programs  by  organizing  DoD  organiza¬ 
tions  to  conduct  the  assessments,  and  it  suggested  appropriate  supported 
and  supporting  roles  of  the  stakeholders.  Bringing  this  approach  to  fru¬ 
ition  would  require  OUSD/P  to  do  the  following  important,  though 
hardly  insurmountable  tasks: 

1.  Assign  supported  and  supporting  roles. 

2.  Delegate  authority  to  specify  assessment  questions  and  data  col¬ 
lection  formats. 

3.  Task  specific  organizations  for  data  collection  and  support. 

4.  Task  specific  organizations  to  conduct  the  assessments,  and 
specifying  the  levels  to  be  addressed. 
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5.  Establish  timelines  and  frequencies  for  assessments,  recognizing 
that  it  will  be  necessary  to  collect  several  years  worth  of  time- 
series  data  in  order  to  conduct  program-level  assessments. 

Despite  the  many  obstacles  to  measuring  security  cooperation 
programs’  direct  contributions  toward  the  end-states  pursued  by  the 
combatant  commands  and  the  national  leadership,  the  suggested 
assessment  framework  presented  here  shows  that  it  is  possible  to  con¬ 
duct  many  security  cooperation  assessments  that  are  entirely  consis¬ 
tent  with  the  spirit  and  intent  of  the  Guidance  for  Employment  of 
the  Force.  The  final  chapter  provides  specific  recommendations  that 
will  enable  OUSD/P  to  move  forward  with  a  new  security  cooperation 
assessment  framework. 


CHAPTER  FIVE 

Conclusions  and  Recommendations 


This  monograph  presents  a  program-level  security  cooperation  assess¬ 
ment  framework  intended  to  enhance  OUSD/P’s  ability  to  make 
informed  resource  and  policy  decisions  at  the  program  level.  It  argues 
that  stakeholder  roles  and  missions,  as  defined  in  the  authorities,  must 
be  clearly  articulated  to  determine  the  appropriate  assessment  responsi¬ 
bilities  for  OUSD/P  program  stakeholders  at  various  levels. 

OUSD/P  is  at  the  center  of  the  security  cooperation  assessment 
pyramid  for  several  key  Title  10  DoD-managed  programs,  as  discussed 
throughout  this  monograph.  OUSD/P  is  also  in  a  unique  position, 
with  the  release  of  the  Guidance  for  Employment  of  the  Force  and 
its  accompanying  assessment  guidance,  to  influence  the  way  in  which 
DoD  thinks  about  and  conducts  its  security  cooperation  assessments. 
For  the  programs  it  manages,  OUSD/P  should  play  the  role  of  assess¬ 
ment  integrator,  rather  than  data  collector  or  assessor,  because  these 
tasks  are  more  suited  to  stakeholders  that  are  directly  involved  in  the 
activities. 

While  the  value  of  security  cooperation  assessments  has  been 
debated,  one  point  stands  true — informed  security  cooperation  deci¬ 
sions  cannot  be  made  without  the  results  of  objective  assessments.  The 
approach  promulgated  in  this  monograph  offers  a  way  to  conceptualize, 
assign  stakeholder  responsibilities,  and  implement  a  new  security  coop¬ 
eration  assessment  framework  at  the  program  level.  This  framework 
was  designed  with  the  interests  of  decisionmakers  clearly  in  mind. 
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Recommendations  for  OUSD/Policy 

We  recommend  that  OUSD/P  and  DSCA  adopt  a  program-level  secu¬ 
rity  cooperation  assessment  framework.  The  following  specific  recom¬ 
mendations  for  implementing  the  assessment  framework  are  laid  out 
in  relation  to  the  five  themes  that  were  articulated  in  the  monograph: 
setting  direction,  defining  assessment  processes,  preparing  stakehold¬ 
ers  for  assessment,  conducting  assessment,  and  explaining  assessment 
results. 

Setting  Direction 

Work  with  key  DoD  stakeholders  to  clarify  program  assessment  responsibil¬ 
ities  in  the  GEF.  OUSD/P  should  engage  key  DoD  stakeholders  in  the 
COCOMs,  services,  combat  support  agencies,  and  defense  agencies  to 
discuss  GEF  program  assessment  requirements  and  to  consider  ways  in 
which  to  implement  the  assessment  approach.  Hosting  program  assess¬ 
ment  seminars,  perhaps  every  other  month,  especially  early  in  the  pro¬ 
cess,  might  be  useful  to  stimulate  learning  and  sharing  of  information 
among  the  various  programs. 

Build  enduring  security  cooperation  goals  and  measurable  program 
objectives.  Enduring  goals  are  important,  but  so  is  having  objectives 
that  are  measurable,  either  from  a  qualitative  or  a  quantitative  perspec¬ 
tive.  Program  managers  are  probably  in  the  best  position  to  develop 
and  recommend  such  objectives,  but  it  would  be  useful  for  OUSD/P 
and  DSCA  to  review  these  objectives  on  a  regular  basis.  These  objec¬ 
tives  should  be  focused  on  more  intermediate,  measurable  program 
milestones,  and  should  be  developed  within  the  broader  context  of 
DoD’s  security  cooperation  programs  and  activities  in  the  specific  part¬ 
ner  countries. 

Clarify  and  standardize  assessment  terminology.  OUSD/P,  as  the 
promulgator  of  the  GEF,  should  use  that  document  to  clarify  key 
assessment  terms,  which  include  program,  program  output ,  measures  of 
progress,  measures  of  effectiveness,  metric,  indicator,  and  focus  areas.  The 
lack  of  agreement  among  stakeholders  on  such  definitions  causes  con¬ 
fusion  in  the  assessment  community.  Adding  these  definitions  to  the 
GEF  would  clarify  the  terminology. 
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Ensure  that  OUSD/P  and  DSCA  are  the  assessment  integrators  for 
programs  they  manage.  Because  OUSD/P  and  DSCA  are  at  the  deci¬ 
sionmaking  level  for  the  programs  they  manage,  these  organizations 
should  serve  as  integrators  of  the  assessments  that  are  conducted  at 
lower  levels.  After- action  reports,  administrative  data,  and  so  on  should 
be  aggregated  by  program  managers,  passed  to  program  assessors,  for¬ 
warded  to  program  reviewers,  and  finally  delivered  to  assessment  inte¬ 
grators  at  the  decisionmaking  level. 

Designing  Assessment  Processes 

Leverage  assessment  processes  where  they  already  exist.  OUSD/P  should 
conduct  a  review  of  assessment  capabilities  for  programs  that  OUSD/P 
and  DSCA  manage.  OUSD/P  and  DSCA  should  consider  ways  to 
ensure  that  stakeholders  are  adequately  resourced  to  perform  their 
assigned  assessment  roles,  particularly  regarding  the  necessary  man¬ 
power,  skills,  training,  and  funding.  Finally,  OUSD/P  should  consider 
providing  more  specific  guidance  on  how  to  assess  and  report  results 
by  program  and  by  stakeholder,  taking  into  account  the  authorities, 
capabilities,  and  objectivity  of  each  stakeholder. 

Within  the  programs,  determine  stakeholder  authorities  and  roles 
among  the five  levels  of  assessment  (e.g,  need for  program,  theory  and  design, 
process  and  implementation,  outcome  and  impact,  and  cost-effectiveness). 
This  monograph  has  discussed  the  importance  of  clarifying  stakeholder 
assessment  authorities,  roles,  and  responsibilities.  It  is  important  for 
all  stakeholders  involved  in  a  given  program  to  understand  their  roles 
and  responsibilities  as  far  as  assessments  are  concerned.  This  discussion 
should  take  place  within  the  context  of  the  program  “team,”  which 
would  include,  for  example,  all  stakeholders  involved  in  a  given  Title 
10  program,  including  the  planners,  resource  managers,  and  those 
involved  in  the  selection  of  partner  countries.  If  program  teams  are  not 
in  a  position  to  clarify  decisionmaking  authorities,  reporting  require¬ 
ments,  and  resourcing  issues,  than  OUSD/P  should  to  act  to  adjudicate 
and  clarify  such  roles  and  responsibilities.  The  basis  of  this  decision 
should  be  the  program  authorities,  the  capabilities  of  the  stakeholders, 
and  the  objectivity  of  stakeholders.  The  stakeholder  roles  should  be 
linked  to  the  five  levels  of  assessment.  OUSD/P  should  encourage  the 
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forming  of  program  teams,  as  mentioned  above,  that  consist  of  all  key 
stakeholders. 

Consider  an  assessment  function  within  OUSD/P  to  assist  with 
the  implementation  of  the  assessment  framework  for  OUSD/P -managed 
programs.  Such  an  assessment  function  could  perform  the  following 
tasks: 

•  Standardize  assessments  across  programs,  where  possible. 

•  Share  insights  across  the  programs  OUSD/P  manages. 

•  Help  ensure  that  assessment  resources  are  available. 

•  Provide  direction  for  data  collection  and  reporting  mechanisms. 

•  Analyze  data  for  security  cooperation  assessments. 

•  Support  assessment  needs  in  theater  at  the  COCOM  and  com¬ 
ponent  levels. 

Program  resources  could  be  used  to  fund  this  assessment  func¬ 
tion,  totaling  approximately  1-3  percent  of  the  program  budget.  This 
function  could  also  deploy  assessment  teams  to  assess  key  effects  in 
high-priority  programs,  such  as  those  under  more  scrutiny  from  Con¬ 
gress.  Where  there  is  an  identified  discrepancy  in  the  data  provided  by 
another  stakeholder,  the  assessment  function  within  OUSD/P  could  be 
deployed  to  perform  an  additional  assessment. 

Preparing  Stakeholders  for  Assessment 

OUSD/Policy  and  OUSD/Personnel  and  Readiness  could  work  with 
Defense  Institute  for  Security  Assistance  Management/DSCA  to  develop  a 
professional  curriculum  for  security  cooperation  assessments.  Such  a  curric¬ 
ulum  would  help  to  prepare  security  cooperation  personnel — including 
data  collectors,  assessors,  reviewers,  and  integrators — for  their  respec¬ 
tive  assessment  responsibilities.  The  training  modules  could  target  mil¬ 
itary  and  civilian  international  affairs  professionals  within  the  DoD. 
The  focus  of  the  core  module  could  be  to  teach  assessment  design,  data 
collection,  evaluation,  and  integration  methods. 

Advanced  coursework  could  include  techniques  for  aggregating 
and  interpreting  assessment  results  to  support  security  cooperation 
decisionmaking  and  analytical  skills  to  support  comparison  and  valu- 
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ation  of  security  cooperation  programs.  Program  resources  could  be 
used  to  fund  this  training. 

Conducting  Assessments 

Specify  stakeholder  data  collection  roles  and  responsibilities.  OUSD/P 
should  work  with  program  managers  to  specify  data  collection  roles 
and  responsibilities  within  each  program.  This  might  be  accomplished 
through  a  few  meetings  led  either  by  OUSD/P  or  by  the  program  man¬ 
ager.  Because  some  of  the  programs  overseen  by  OSD  are  actually  ini¬ 
tiatives  or  collections  of  programs  (i.e.,  the  Warsaw  Initiative  Fund), 
programs  within  those  initiatives  should  be  considered  individually  for 
assessment  purposes. 

Explore  external  indicators  such  as  socioeconomic  indicators.  Con¬ 
sider  research  indicators  used  by  the  State  Department,  USAID,  the 
World  Bank,  implementing  partners,  embassies,  and  nongovernmental 
organizations  as  sources  of  data.  Such  indicators  help  to  interject  addi¬ 
tional  objectivity  into  the  assessment  framework. 

Standardize  assessment  questions  within  and  across  programs, 
as  appropriate.  Consider  a  time-phased  data  collection  approach  in 
which  standard  questions  are  asked  over  time  to  compare  and  contrast 
results.  Fiere  again,  program  managers  should  be  primarily  responsi¬ 
ble  for  developing  such  questions,  but  OUSD/P  and  DSCA,  as  the 
assessment  integrators,  should  validate  these  questions  and  encourage 
program  managers  to  share  them,  once  they  have  been  developed  and 
validated. 

Develop  a  feedback  loop  for  “setting  direction.  ”  Assessment  results 
should  be  used  to  provide  feedback  that  can  be  used  to  refine  the  “set¬ 
ting  direction”  phase  within  the  assessment  process.  As  the  ultimate 
proponent,  policymaker,  and  assessment  integrator,  OUSD/P  would 
be  in  the  position  to  interpret  the  result  and  to  make  changes  to  the 
guidance,  when  required.  Such  changes  might  include  updates  to  the 
GEF  assessment  guidance  to  the  COCOMs,  services,  combat  support 
agencies,  and  defense  agencies. 

Consider  a  pilot  program  to  test  the  assessment framework.  OUSD/P 
should  consider  conducting  a  pilot  program  to  test  the  implementa¬ 
tion  of  the  assessment  framework.  Some  programs  under  OUSD/P’s 
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management  purview  are  under  greater  scrutiny  from  Congress  than 
others.  OUSD/P  should  consider  a  pilot  program  where  there  is  a  more 
immediate  need  to  know  the  results,  such  as  for  1206  Global  Train  and 
Equip.  A  second  pilot,  perhaps  with  a  program  that  has  a  longer  his¬ 
tory  and  richer  data,  should  also  be  considered  to  thoroughly  test  the 
assessment  framework.  The  Warsaw  Initiative  Fund  and  its  subsequent 
programs  would  be  a  good  candidate  program  to  consider. 

Explaining  Assessment  Results 

Develop  clearer  linkages  between  assessment  and  planning.  OUSD/P 
should  establish  a  mechanism  to  enable  assessment  and  planning  staffs 
to  better  work  together  to  revise  plans,  establish  planning  and  assess¬ 
ment  benchmarks,  and  assess  progress.  Encouraging  discussion  is  a  first 
step — the  results  of  such  discussions  should  be  shared  with  OUSD/P, 
perhaps  within  the  auspices  of  a  GEF  program  assessment  meeting. 

Use  results  to  inform  decisions  about  programs.  Program  assessment 
results  are  important  for  making  informed  decisions  as  to  whether  to 
continue,  cut,  expand,  or  diminish  programs  and  their  subsequent 
activities,  where,  and  by  how  much.  Such  results  can  be  very  useful  in 
defending  program  decisions  that  have  been  made  to  external  stake¬ 
holders,  such  as  Congress,  or  to  respective  partner  countries. 

Identify  examples  where  multiple,  coordinated  programs  have 
achieved  desired  effects.  OUSD/P  and  DSCA,  as  assessment  integra¬ 
tors,  should  strive  to  identify  examples  of  cross-program  coordination 
and  their  benefits  from  an  assessment  perspective.  It  is  desirable,  from 
a  strategy  and  resourcing  perspective,  for  program  managers  to  work 
together  to  reinforce  key  concepts  in  the  partner  countries  and  thus 
achieve  the  same  desirable  effects.  It  is  important  to  capture  the  lessons 
and  best  practices  in  which  two  or  more  programs  are  coordinated 
in  this  way.  Such  benefits  of  effective  coordination  should  be  shared 
with  other  program  stakeholders  and  with  external  stakeholders,  such 
as  Congress. 

Consider  using  the  Global  Theater  Security  Cooperation  Manage¬ 
ment  Information  System  (TSCMIS)  as  a  repository  for  program  assess¬ 
ments.  Currently  there  is  no  one  place  where  program  assessment 
information  is  collected  and  disseminated.  Thus,  in  planning  for  the 
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new  Global  TSCMIS,  which  will  link  the  various  COCOM  and 
service  theater  security  cooperation  data  networks,  DSCA  should 
consider  how  to  incorporate  program-level  assessments  into  this  new, 
integrated  system  in  a  way  that  will  support  the  GEF-directed  pro¬ 
gram-level  assessments.  Program  assessment  information  should  be 
collected  and  disseminated  on  the  Global  TSCMIS  system  when  it 
is  built.  Having  such  a  centralized  data  collection  and  dissemination 
system  will  enable  OUSD/P  and  DSCA  to  better  communicate  the 
results  of  the  assessment  process  to  internal  and  external  stakeholders. 


Conclusion 

The  assessment  framework  proposed  in  this  monograph  is  intended 
to  move  the  security  cooperation  assessment  process  away  from  self- 
assessments  by  program  managers  and  to  expand  on  immediate  assess¬ 
ments  by  COCOMs.  The  goal  is  a  process  that  is  consistent  enough 
across  programs  to  be  convincing — to  OSD  leaders  and  to  Congress — 
but  flexible  enough  to  account  for  the  differences  across  programs.  In 
particular,  the  framework  is  intended  to  focus  on  decisions  and  on  the 
requirements  of  decisionmakers.  It  recognizes  both  the  range  of  stake¬ 
holders  in  any  given  security  cooperation  program  and  the  fact  that 
their  interests  often  extend  beyond  their  formal  authorities.  It  seeks  to 
match  those  who  participate  in  the  assessment  process  with  the  capa¬ 
bilities  and  interests  of  the  stakeholders.  Finally,  it  strives  to  provide 
OUSD/P  a  way  to  look  across  all  security  cooperation  programs  in  its 
efforts  to  assure  policymakers  and  the  public  that  our  nation  receives 
the  most  security  possible  from  its  security  cooperation  efforts. 

Although  the  focus  of  this  monograph  is  on  the  program  level 
of  security  cooperation,  as  explained  in  the  introduction,  we  stress  the 
need  for  a  national-level  assessment  that  synthesizes  the  evaluations  of 
all  security  cooperation  programs.  A  national-level  assessment  mecha¬ 
nism  is  necessary  to  underpin  decisions  concerning  resource  and  pro¬ 
gram  increases,  decreases,  and  reallocations.  This  monograph  recom¬ 
mends  an  initial  step  in  that  direction  by  suggesting  an  integrating 
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role  for  OUSD/P.  However,  that  function  falls  somewhat  short  of  the 
comprehensive  national-level  integrated  security  cooperation  assess¬ 
ment  that  will  ultimately  be  required.  Such  an  assessment  should  com¬ 
port  with  the  security  cooperation  priorities  delineated  in  the  GEF. 


APPENDIX  A 


Program  Descriptions 


This  appendix  provides  a  descriptive  overview  of  the  six  OUSD/P- 
managed  programs  listed  in  Chapter  One. 


Warsaw  Initiative  Fund 

Activities 

The  Warsaw  Initiative  Fund  supports  developing  NATO  Partnership 
for  Peace  member  countries  in  reforming  their  defense  institutions, 
improving  interoperability  with  the  United  States  and  NATO,  and 
integrating  further  with  NATO.  The  OUSD/P  Partnership  Strategy 
has  ten  focus  areas  for  all  WIF  efforts.  In  order  of  priority,  they  are  the 
following: 

1.  Defense  policy  and  strategy 

2.  Human  resource  management 

3.  Democratic  control  of  armed  forces 

4.  Defense  planning,  budgeting,  and  resource  management 

5.  Logistics 

6.  Peacekeeping  operations:  conceptual,  planning,  and  operational 
aspects  of  peacekeeping 

7.  Professional  defense  and  military  education:  education,  train¬ 
ing,  and  doctrine 

8.  Emergency  planning/consequence  management 

9.  Border  security  and  control 
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10.  English  language  (although  WIF  may  not  be  used  to  support 
language  training  programs). 


Key  Stakeholders 

OUSD/P  Partnership  Strategy  provides  WIF  policy  oversight  and 
DSCA  is  responsible  for  program  management.  The  Office  of  the  Assis¬ 
tant  Secretary  of  Defense  for  International  Security  Affairs/  Europe  and 
NATO  and  the  Office  of  the  Assistant  Secretary  of  Defense  for  Asia- 
Pacific  Security  Affairs/Central  Asia  provide  regional  policy  guidance 
and  input  into  programs.  All  Partnership  for  Peace  member  countries 
are  within  the  EUCOM  and  CENTCOM  areas  of  responsibility.  These 
COCOMs  are  responsible  for  developing  and  implementing  the  major¬ 
ity  of  WIF  activities  within  their  respective  areas  of  responsibility. 

Authorities 

WIF  is  governed  by  10  U.S.C.  168,  1051,  and  2010  (Appendix  C)  and 
PL  108-375,  Section  1224.  These  authorities  regulate  WIF  funding  for 
bilateral  or  regional  cooperation  programs;  participation  of  developing 
countries  in  combined  exercises;  and  military-to-military  contacts  and 
comparable  activities.  WIF  can  be  used  to  fund  costs  associated  with 
activities  and  travel/per  diem. 

Resources 

In  fiscal  year  (FY)  2008,  WIF  was  funded  at  $28.75  million.  WIF  is 
funded  from  within  the  Defense-Wide  Operations  and  Maintenance 
account  in  the  defense  budget.  Warsaw  Initiative  Funds  are  one-year 
funds.  Funds  were  administered  by  the  Joint  Staff  from  FYs  1995  to 
1997,  by  OUSD/P  in  FY  1998,  and  by  DSCA  since  FY  1999. 

Current  Assessment  Processes 

The  Warsaw  Initiative  Fund’s  objectives  are  based  on  the  original  goals 
of  the  Partnership  for  Peace  Framework  Document.  WIF’s  objectives 
have  been  altered  slightly  over  time  in  response  to  changes  in  Partner¬ 
ship  for  Peace  and  NATO.  WIF’s  three  objectives  are  the  following: 


Program  Descriptions  79 


1.  Advance  partner  Defense  Institution  Building  (DIB)/defense 
reform  in  partner  countries. 

2.  Enhance  U.S./NATO-partner  interoperability  to  enhance  part¬ 
ner  contributions  to  coalition  operations. 

3.  Support  partner  integration  with  NATO. 

CENTCOM  and  EUCOM  are  responsible  for  developing  and 
implementing  the  majority  of  WIF  activities  within  their  respective 
areas  of  responsibility.  However,  in  FY  2006,  OUSD/P  launched  the 
DIB  concept,  which  is  designed  to  streamline  all  WIF  (and  other  pro¬ 
gram)  defense  reform  efforts  in  order  to  focus  priorities  and  funding 
and  to  avoid  gaps.  OUSD/P  Partnership  Strategy  and  DSCA  have 
tasked  the  Center  for  Civil-Military  Relations  to  provide  the  analyti¬ 
cal  support  and  subject  matter  expertise  needed  to  execute  the  DIB 
program.  The  DIB  program  typically  consists  of  a  policy  assessment 
survey,  roadmap  development,  program  implementation,  and  follow¬ 
up  policy  and  programmatic  assessment  visits.  DIB  surveys  are  con¬ 
ducted  over  the  course  of  a  week  to  capture  data  and  the  real-time 
status  of  a  partner  country’s  defense  institutions  in  the  ten  areas  identi¬ 
fied  above.  The  DIB  Roadmap  is  a  living  document,  developed  from 
these  surveys,  that  contains  an  action  plan  for  the  partner  country  to 
pursue  in  order  to  improve  its  defense  and  security-sector  institutions. 
Using  this  roadmap,  a  menu  of  activities  is  developed  and  implemented 
to  assist  the  partner  country  in  continuing  improvement  of  its  defense 
and  security  institutions. 


National  Guard  State  Partnership  Program 

Activities 

The  National  Guard  provides  unique  capacity-building  capabilities 
to  combatant  commanders  and  U.S.  ambassadors  via  58  comprehen¬ 
sive  partnerships  between  U.S.  states  and  partner  nations.  The  State 
Partnership  Program  directly  supports  the  broad  U.S.  national  inter¬ 
ests  and  security  cooperation  goals  by  engaging  partner  nations  via 
military,  sociopolitical,  and  economic  contact  at  the  local,  state,  and 
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national  levels.  States  and  their  partners  participate  in  a  broad  range 
of  strategic  security  cooperation  activities,  including  homeland  defense 
and  security;  disaster  response  and  mitigation;  consequence  and  crisis 
management;  interagency  cooperation;  border,  port,  and  aviation  secu¬ 
rity;  combat  medical  internships,  and  bilateral  familiarization  events 
that  lead  to  training  and  exercise  opportunities. 

Key  Stakeholders 

All  state  activities  are  coordinated  through  the  combatant  command¬ 
ers,  U.S.  ambassadors’  country  teams,  and  other  agencies,  as  appropri¬ 
ate,  to  ensure  that  National  Guard  cooperation  is  tailored  to  meet  U.S. 
and  international  partners’  objectives. 

Authorities 

32  U.S.C.  and  the  National  Defense  Authorization  Act  of  1993  (annual 
renewal  needed). 

Resources 

State  Partnership  Program  activities  are  funded  largely  through 
COCOM  theater  command  activities  or  state  funds. 

Current  Assessment  Practices 

The  National  Guard  Bureau/International  Affairs  (NGB-IA)  cur¬ 
rently  assesses  program  efficiency  by  tracking  all  SPP  events  through 
desk  officers  at  NGB-IA  or  through  SPP  liaison  officers  embedded 
in  the  COCOM.  NGB-IA  is  also  developing  measures  of  effective¬ 
ness  concerning  goals  and  end-states  for  the  program.  These  goals 
include  building  partnership  capacity  to  deter,  prevent,  and  prepare;  to 
respond  and  recover;  to  support  partners’  defense  reform  and  profes¬ 
sional  development;  and  to  facilitate  enduring  broad-spectrum  security 
relationships. 
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Global  Train  and  Equip  (1206)  Program 

Activities 

The  Global  Train  and  Equip  Program  enables  DoD  to  conduct 
capacity-building  programs  with  foreign  military  partners  to  improve 
their  ability  to  conduct  counterterrorist  operations  or  to  support  mili¬ 
tary  and  stability  operations  in  areas  in  which  the  U.S.  armed  forces 
are  also  deployed. 

Key  Stakeholders 

Office  of  the  Secretary  of  Defense,  State  Department,  Political- 
Military  Bureau,  geographic  combatant  commands,  and  component 
commands. 

Authorities 

PL  109-163,  Section  1206  (subject  to  annual  renewal). 

Resources 

$300  million  authorized  from  within  Defense-Wide  Operations  and 
Maintenance  funding. 

Current  Assessment  Practices 

Currently,  each  COCOM  is  required  to  provide  an  assessment  of  the 
effects  of  1206  activities  in  its  theater.  For  example,  a  recent  indepen¬ 
dent  assessment  was  conducted  by  the  Center  for  Naval  Analyses  on 
the  operational  effects  of  a  1206  Train  and  Equip  project  in  Lebanon. 


The  Regional  Centers 

Activities 

The  Regional  Centers  build  and  sustain  a  network  of  current  and  future 
security  leaders  who  share  common  values  and  perspectives,  strive  to 
increase  their  national  capacity  to  meet  internal  security  needs  while 
contributing  to  the  security  of  others,  and  promote  greater  interna¬ 
tional  cooperation.  The  centers  focus  on  network-building  through 
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resident  executive  education  and  in-region  conferences,  seminars,  and 
workshops.  Focusing  on  regional  audiences  in  a  global  context,  they 
widen  perspectives  and  enhance  critical  thinking.  The  Regional  Cen¬ 
ters  are  recognized  as  facilitators  of  open  exchanges  of  ideas. 

Key  Stakeholders 

COCOMs,  DSCA,  U.S.  ambassadors’  country  teams,  and  OSD. 

Authorities 

10  U.S.C.  184,  1050,  and  1051;  PL  109-364,  Section  904;  and  DoD 
Directive  5200.41. 

Resources 

The  FY  2008  total  was  approximately  $88  million,  according  to  Pro¬ 
gram  Objective  Memorandum  10-15,  which  includes  the  five  cen¬ 
ters,  the  Global  Center  in  Monterey,  and  administrative  expenses  for 
DSCA. 

Current  Assessment  Practices 

Each  Regional  Center  is  devising  an  assessment  of  its  programs.  For 
example,  the  Marshall  Center  has  a  fairly  extensive  student  and  alumni 
survey  system  to  help  gauge  program  efficiency  as  well  as  effectiveness 
over  a  longer  time  frame. 


Combating  Terrorism  Fellowship  Program 

Activities 

The  Regional  Defense  Combating  Terrorism  Fellowship  Program  is  a 
security  cooperation  tool  that  provides  education  to  international  secu¬ 
rity  personnel  as  part  of  the  global  effort  to  combat  terrorism  while 
reducing  stress  on  U.S.  forces.  To  achieve  these  goals,  CTFP  provides 
the  geographic  combatant  commanders  with  resources  to  foster  regional 
cooperation  and  to  professionalize  and  expand  foreign  capabilities  to 
combat  terrorism.  The  program  provides  targeted  combating-terrorism 
education  to  key  senior  and  mid-level  military  officials,  ministry  of 
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defense  civilians,  and  other  foreign  government  security  officials.  Per¬ 
sonnel  recommended  for  CTFP  engagement  by  the  combatant  com¬ 
mands  are  expected  to  have  a  positive  impact  on  their  country’s  ability 
to  cooperate  with  the  United  States  in  the  war  on  terrorism. 

Key  Stakeholders 

The  Assistant  Secretary  of  Defense  for  Global  Security  Affairs  pro¬ 
vides  policy  oversight.  DSCA  provides  financial  management.  CTFP 
requires  approval  from  the  chiefs  of  missions,  combatant  commanders, 
and  OUSD/P  prior  to  any  event  or  engagement. 

Authorities 

CTFP  is  permanently  authorized  in  10  U.S.C.  2249c. 

Resources 

The  2007  National  Defense  Authorization  Act  authorized  a  $25  million 
budget  of  appropriated  funds.  There  is  a  legislative  proposal  to  increase 
the  authorization  to  $35  million.  CTFP  funds  come  from  DSCA  and 
Defense-Wide  Operations  and  Maintenance  funding. 

Current  Assessment  Practices 

CTFP  is  required  to  submit  a  report  to  Congress  that  accounts  for  the 
expenditure  of  appropriated  funds  during  each  fiscal  year.  This  report 
must  include  the  countries  that  received  CTFP  funding,  the  total 
amount  of  funding  provided  for  each  country,  and  the  events/courses 
attended  by  the  foreign  officers  and  officials.  The  report  also  includes 
an  assessment  of  the  effectiveness  of  the  program  and  a  review  of 
efforts  to  improve  it.  In  FY  2007,  CTFP  sponsored  or  funded  multiple 
combating-terrorism  educational  events  that  involved  2,737  secu¬ 
rity  personnel  from  115  countries,  all  with  the  goal  of  meeting  the 
combating-terrorism  needs  identified  by  the  regional  combatant 
commanders. 
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Overseas  Humanitarian,  Disaster,  and  Civic  Aid  Program 

Activities 

The  Foreign  Disaster  Assistance  program  enables  DoD  to  assist  coun¬ 
tries  in  their  response  to  disasters  when  necessary  to  prevent  the  loss  of 
life.  Services  and  supplies,  logistical  support,  search  and  rescue,  medi¬ 
cal  evacuation,  and  refugee  assistance  may  be  provided.  The  goals  of 
the  program  include  supporting  U.S.  foreign  policy  and  national  secu¬ 
rity  goals  of  regional  stability,  promotion  of  democracy,  and  economic 
development. 

Key  Stakeholders 

COCOMs,  U.S.  ambassadors’  country  teams,  DSCA,  and  OSD. 

Authorities 

10  U.S.C.  402,  404,  2557,  and  2561. 

Resources 

Approximately  $63  million  was  obligated  for  humanitarian  assistance 
during  FY  2006.  More  than  $50  million  of  this  was  used  to  provide 
disaster  relief  to  Pakistan. 

Current  Assessment  Practices 

Currently,  DoD  does  not  have  an  assessment  of  humanitarian  assis¬ 
tance  programs  beyond  detailing  which  projects  are  funded  and  where 
they  are  executed  each  year. 


APPENDIX  B 


Overview  of  Main  Points  from  October  2008 
Assessment  Workshop 


This  appendix  provides  an  overview  of  the  key  points  from  a  follow- 
on  assessment  workshop  held  in  October  2008.  The  purpose  of  the 
workshop  was  to  discuss  how  an  assessment  function  within  OUSD/P 
Partnership  Strategy  might  be  developed.  The  workshop  included  par¬ 
ticipants  from  OSD  PA&E,  DSCA  Enterprise  Services,  and  OUSD/P 
Partnership  Strategy,  in  addition  to  several  program  managers  and 
RAND  analysts. 

The  discussion  began  with  an  overview  of  program  assessment 
requirements  for  OUSD/P  security  cooperation  programs.  This 
included  a  new  requirement  in  Section  1237  of  the  2009  National 
Defense  Authorization  Act  for  an  annual  Report  on  Utilization  of  Cer¬ 
tain  Global  Partnership  Authorities,  including 

•  an  assessment  of  the  impact  of  the  assistance  provided  under  the 
rubric  of  “Building  Partnership”  with  respect  to  each  country 
receiving  assistance 

•  a  description  of  the  processes,  if  any,  used  by  the  Department  of 
Defense  and  the  Department  of  State  to  evaluate  the  success  of 
each  project  so  funded  after  its  completion 

•  an  assessment  of  the  utility  of  the  authorities,  any  gaps  in  such 
authorities,  and  the  feasibility  and  advisability  of  continuing  such 
authorities  beyond  their  current  dates  of  expiration. 

The  requirement  is  the  latest  in  a  U.S .  government  and  DoD  effort 
to  provide  a  clearer  understanding  of  U.S.  foreign  assistance  and  secu- 
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rity  cooperation  programs.  It  further  underscored  for  the  workshop 
participants  that  a  function  is  needed  to  help  programs,  COCOMs, 
services,  and  ultimately  OSD  assess  the  effectiveness  of  security  coop¬ 
eration  programs. 

Several  policy  improvements  are  also  under  way  that  could  help  to 
improve  the  management  of  security  cooperation  resources.  For  exam¬ 
ple,  OUSD/P  formed  a  Building  Partnership  Capability  Portfolio  to  try 
to  manage  the  funding  of  various  security  cooperation  programs  across 
DoD  program  elements.  Portfolio  managers  will  draw  on  information 
from  both  OUSD/P  and  Joint  Staff/J-5  to  help  determine  the  cost- 
effectiveness  of  BPC  programs.  OSD  PA&E  is  also  using  a  new  data¬ 
base  of  “SNaP”  (Select  and  Native  Programming  data  input  system) 
reports  on  subprogram  elements  to  calculate  the  resource  expenditures 
of  over  300  programs  that  help  build  partner  capacity.  As  the  data¬ 
base  is  populated,  it  will  become  easier  for  OSD  PA&E  to  manage  the 
resources  devoted  to  this  growing  focus  area. 

Workshop  participants  generally  favored  the  formation  of  an 
OUSD/P  assessment  function  composed  of  internal  experts  who  could 
help  program  managers,  COCOMs,  and  OUSD/P  stakeholders  to 
integrate  existing  assessments  and  develop  templates  for  additional 
assessments.  Participants  agreed  that  such  an  assessment  function 
would  need  a  clear  mandate  and  defined  roles  and  responsibilities  that 
included  access  to,  and  influence  over,  security  cooperation  planning 
and  budgetary  processes.  Furthermore,  the  function  should  help  estab¬ 
lish  policies  and  procedures  for  conducting  program  assessments. 

The  assessment  function  should  include  personnel  with  a  mix  of 
technical  skills  and  management  expertise,  equipment,  and  software. 
Staff  members  should  have  access  to  training  and  professional  develop¬ 
ment.  The  evaluation  group  will  also  require  a  change  management 
plan  and  communications  plan  to  help  coordinate  the  launch  of  this 
new  assessment  endeavor. 

The  final  session  centered  on  two  issues:  the  actual  cost  of  assess¬ 
ment  and  the  cost  of  the  proposed  assessment  function.  Is  it  possible 
to  estimate  the  cost  of  assessing  programs?  What  kind  of  contracting 
vehicles  exist?  Is  it  possible  to  share  costs?  What  does  it  cost  to  train 
evaluators?  The  participants,  who  included  representatives  from  the 
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OSD/Comptroller’s  office,  stated  that  the  base  cost  for  core  personnel 
and  systems  is  probably  greater  for  a  new  program,  such  as  1206  Global 
Train  and  Equip,  than  it  is  for  more  established  programs,  such  as  the 
Regional  Centers. 

Unfortunately,  there  is  no  easy  way  to  determine  how  much  a  pro¬ 
gram  evaluation  will  cost  until  policies  and  procedures  are  established 
and  work  can  begin.  This  leads  to  a  paradox:  OUSD/P  requires  fund¬ 
ing  to  establish  a  strategic  vision  and  plan  for  the  assessment  function, 
but  program  managers  are  resistant  to  offering  funding  unless  they 
know  what  the  policies  and  procedures  will  be.  A  way  ahead  could  be 
to  choose  a  “champion”  program  that  is  led  by  a  well-respected  man¬ 
ager  and  is  already  quite  far  along  in  its  own  assessment  capacity.  This 
approach  has  two  main  advantages:  It  helps  the  new  assessment  func¬ 
tion  to  hone  policies  and  procedures,  and  it  demonstrates  the  potential 
value  of  such  a  function  to  key  stakeholders  associated  with  OUSD/P- 
managed  security  cooperation  programs. 
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